You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/06/15 06:02:41 UTC
[JENKINS] Lucene-Solr-4.x-MacOSX (64bit/jdk1.6.0) - Build # 547 -
Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/547/
Java: 64bit/jdk1.6.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
1 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 119 from http://127.0.0.1:52988/m_t/collection1lastClient and got 80 from http://127.0.0.1:53016/m_t/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 119 from http://127.0.0.1:52988/m_t/collection1lastClient and got 80 from http://127.0.0.1:53016/m_t/collection1
at __randomizedtesting.SeedInfo.seed([7C6E80877E7D058:8620661000B8B064]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:963)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:137)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:680)
Build Log:
[...truncated 9060 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4:junit4] 2> 2364862 T4278 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /m_t/
[junit4:junit4] 2> 2364895 T4278 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solrtest-ChaosMonkeySafeLeaderTest-1371267304849
[junit4:junit4] 2> 2364898 T4278 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 2364899 T4279 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 2364904 T4278 oasc.ZkTestServer.run start zk server on port:52981
[junit4:junit4] 2> 2364935 T4278 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2364958 T4285 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@28cf7112 name:ZooKeeperConnection Watcher:127.0.0.1:52981 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2364959 T4278 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2364960 T4278 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 2364981 T4278 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2364987 T4287 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@750a75c name:ZooKeeperConnection Watcher:127.0.0.1:52981/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2364987 T4278 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2364988 T4278 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 2364998 T4278 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 2365011 T4278 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 2365018 T4278 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 2365026 T4278 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 2365027 T4278 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 2365060 T4278 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 2365061 T4278 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 2365070 T4278 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 2365070 T4278 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 2365084 T4278 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 2365087 T4278 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 2365094 T4278 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 2365095 T4278 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 2365103 T4278 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 2365105 T4278 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 2365113 T4278 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 2365114 T4278 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 2365124 T4278 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 2365125 T4278 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 2365134 T4278 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 2365135 T4278 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 2365145 T4280 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x13f45e68d9e0001, likely client has closed socket
[junit4:junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
[junit4:junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:680)
[junit4:junit4] 2>
[junit4:junit4] 2> 2365653 T4278 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 2365664 T4278 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52984
[junit4:junit4] 2> 2365669 T4278 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 2365669 T4278 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 2365669 T4278 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371267305098
[junit4:junit4] 2> 2365670 T4278 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371267305098/solr.xml
[junit4:junit4] 2> 2365670 T4278 oasc.CoreContainer.<init> New CoreContainer 1904445276
[junit4:junit4] 2> 2365671 T4278 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371267305098/'
[junit4:junit4] 2> 2365671 T4278 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371267305098/'
[junit4:junit4] 2> 2365873 T4278 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 2365875 T4278 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 2365876 T4278 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 2365876 T4278 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 2365877 T4278 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 2365878 T4278 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 2365878 T4278 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 2365879 T4278 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 2365879 T4278 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 2365881 T4278 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 2365950 T4278 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 2365951 T4278 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52981/solr
[junit4:junit4] 2> 2365952 T4278 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 2365966 T4278 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2365968 T4298 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a08ad35 name:ZooKeeperConnection Watcher:127.0.0.1:52981 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2365970 T4278 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2365974 T4278 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 2366034 T4278 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2366041 T4300 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7631f46b name:ZooKeeperConnection Watcher:127.0.0.1:52981/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2366041 T4278 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2366045 T4278 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 2366060 T4278 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 2366073 T4278 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 2366079 T4278 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52984_m_t
[junit4:junit4] 2> 2366081 T4278 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52984_m_t
[junit4:junit4] 2> 2366091 T4278 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 2366103 T4278 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 2366109 T4278 oasc.Overseer.start Overseer (id=89867374091436035-127.0.0.1:52984_m_t-n_0000000000) starting
[junit4:junit4] 2> 2366116 T4278 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 2366128 T4302 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 2366135 T4278 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 2366142 T4278 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 2366149 T4278 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 2366161 T4301 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 2366167 T4303 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 2366167 T4303 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 2367676 T4301 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2367678 T4301 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52984_m_t",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52984/m_t"}
[junit4:junit4] 2> 2367678 T4301 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 2367680 T4301 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 2367697 T4300 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 2368176 T4303 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371267305098/collection1
[junit4:junit4] 2> 2368177 T4303 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 2368180 T4303 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 2368181 T4303 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 2368189 T4303 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371267305098/collection1/'
[junit4:junit4] 2> 2368193 T4303 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371267305098/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 2368194 T4303 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371267305098/collection1/lib/README' to classloader
[junit4:junit4] 2> 2368307 T4303 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 2368426 T4303 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 2368429 T4303 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 2368437 T4303 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 2369634 T4303 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 2369661 T4303 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 2369666 T4303 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 2369708 T4303 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 2369717 T4303 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 2369726 T4303 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 2369729 T4303 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 2369729 T4303 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 2369730 T4303 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 2369733 T4303 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 2369733 T4303 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 2369734 T4303 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 2369734 T4303 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371267305098/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/control/data/
[junit4:junit4] 2> 2369734 T4303 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7a8eae04
[junit4:junit4] 2> 2369735 T4303 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 2369737 T4303 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/control/data
[junit4:junit4] 2> 2369738 T4303 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/control/data/index/
[junit4:junit4] 2> 2369739 T4303 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 2369740 T4303 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/control/data/index
[junit4:junit4] 2> 2369749 T4303 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 2369749 T4303 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 2369759 T4303 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 2369761 T4303 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 2369763 T4303 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 2369764 T4303 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 2369765 T4303 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 2369765 T4303 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 2369766 T4303 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 2369767 T4303 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 2369767 T4303 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 2369778 T4303 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 2369790 T4303 oass.SolrIndexSearcher.<init> Opening Searcher@41feef9 main
[junit4:junit4] 2> 2369792 T4303 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 2369793 T4303 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 2369798 T4304 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@41feef9 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 2369801 T4303 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 2369801 T4303 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52984/m_t collection:control_collection shard:shard1
[junit4:junit4] 2> 2369803 T4303 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 2369827 T4303 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 2369835 T4303 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 2369835 T4303 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 2369835 T4303 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52984/m_t/collection1/
[junit4:junit4] 2> 2369836 T4303 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 2369836 T4303 oasc.SyncStrategy.syncToMe http://127.0.0.1:52984/m_t/collection1/ has no replicas
[junit4:junit4] 2> 2369836 T4303 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52984/m_t/collection1/
[junit4:junit4] 2> 2369836 T4303 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 2370735 T4301 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2370757 T4300 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 2370810 T4303 oasc.ZkController.register We are http://127.0.0.1:52984/m_t/collection1/ and leader is http://127.0.0.1:52984/m_t/collection1/
[junit4:junit4] 2> 2370811 T4303 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52984/m_t
[junit4:junit4] 2> 2370811 T4303 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 2370811 T4303 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 2370812 T4303 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 2370815 T4303 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2370818 T4278 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4] 2> 2370818 T4278 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 2370820 T4278 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 2370875 T4278 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 2370890 T4278 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2370892 T4307 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@70638cdf name:ZooKeeperConnection Watcher:127.0.0.1:52981/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2370893 T4278 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2370896 T4278 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 2370903 T4278 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 2371446 T4278 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 2371490 T4278 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52988
[junit4:junit4] 2> 2371493 T4278 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 2371496 T4278 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 2371497 T4278 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371267310856
[junit4:junit4] 2> 2371498 T4278 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371267310856/solr.xml
[junit4:junit4] 2> 2371498 T4278 oasc.CoreContainer.<init> New CoreContainer 216873528
[junit4:junit4] 2> 2371499 T4278 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371267310856/'
[junit4:junit4] 2> 2371500 T4278 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371267310856/'
[junit4:junit4] 2> 2371697 T4278 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 2371698 T4278 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 2371699 T4278 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 2371699 T4278 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 2371700 T4278 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 2371700 T4278 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 2371701 T4278 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 2371702 T4278 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 2371702 T4278 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 2371703 T4278 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 2371756 T4278 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 2371758 T4278 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52981/solr
[junit4:junit4] 2> 2371759 T4278 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 2371760 T4278 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2371768 T4318 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@71c0 name:ZooKeeperConnection Watcher:127.0.0.1:52981 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2371769 T4278 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2371774 T4278 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 2371820 T4278 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2371826 T4320 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e405285 name:ZooKeeperConnection Watcher:127.0.0.1:52981/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2371826 T4278 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2371839 T4278 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 2372285 T4301 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2372289 T4301 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52984_m_t",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52984/m_t"}
[junit4:junit4] 2> 2372302 T4300 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 2372303 T4320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 2372313 T4307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 2372850 T4278 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52988_m_t
[junit4:junit4] 2> 2372855 T4278 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52988_m_t
[junit4:junit4] 2> 2372860 T4320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 2372866 T4320 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 2372872 T4300 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 2372872 T4300 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2372882 T4321 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 2372882 T4321 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 2372885 T4307 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 2372885 T4307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2373817 T4301 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2373824 T4301 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52988_m_t",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52988/m_t"}
[junit4:junit4] 2> 2373824 T4301 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 3
[junit4:junit4] 2> 2373825 T4301 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 2373837 T4300 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2373837 T4320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2373837 T4307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2373886 T4321 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371267310856/collection1
[junit4:junit4] 2> 2373886 T4321 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 2373888 T4321 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 2373888 T4321 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 2373891 T4321 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371267310856/collection1/'
[junit4:junit4] 2> 2373893 T4321 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371267310856/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 2373914 T4321 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371267310856/collection1/lib/README' to classloader
[junit4:junit4] 2> 2374008 T4321 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 2374164 T4321 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 2374168 T4321 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 2374176 T4321 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 2375431 T4321 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 2375452 T4321 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 2375458 T4321 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 2375497 T4321 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 2375508 T4321 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 2375541 T4321 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 2375544 T4321 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 2375548 T4321 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 2375549 T4321 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 2375553 T4321 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 2375554 T4321 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 2375554 T4321 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 2375555 T4321 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371267310856/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty1/
[junit4:junit4] 2> 2375555 T4321 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7a8eae04
[junit4:junit4] 2> 2375556 T4321 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 2375557 T4321 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty1
[junit4:junit4] 2> 2375559 T4321 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty1/index/
[junit4:junit4] 2> 2375560 T4321 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 2375561 T4321 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty1/index
[junit4:junit4] 2> 2375569 T4321 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 2375570 T4321 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 2375575 T4321 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 2375576 T4321 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 2375577 T4321 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 2375578 T4321 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 2375579 T4321 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 2375579 T4321 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 2375580 T4321 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 2375587 T4321 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 2375591 T4321 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 2375602 T4321 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 2375615 T4321 oass.SolrIndexSearcher.<init> Opening Searcher@36037b7a main
[junit4:junit4] 2> 2375616 T4321 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 2375617 T4321 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 2375623 T4322 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@36037b7a main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 2375628 T4321 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 2375629 T4321 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52988/m_t collection:collection1 shard:shard1
[junit4:junit4] 2> 2375630 T4321 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 2375655 T4321 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 2375662 T4321 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 2375662 T4321 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 2375663 T4321 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52988/m_t/collection1/
[junit4:junit4] 2> 2375663 T4321 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 2375663 T4321 oasc.SyncStrategy.syncToMe http://127.0.0.1:52988/m_t/collection1/ has no replicas
[junit4:junit4] 2> 2375664 T4321 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52988/m_t/collection1/
[junit4:junit4] 2> 2375664 T4321 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 2376897 T4301 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2376922 T4300 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2376923 T4320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2376922 T4307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2376970 T4321 oasc.ZkController.register We are http://127.0.0.1:52988/m_t/collection1/ and leader is http://127.0.0.1:52988/m_t/collection1/
[junit4:junit4] 2> 2376971 T4321 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52988/m_t
[junit4:junit4] 2> 2376971 T4321 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 2376971 T4321 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 2376972 T4321 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 2376975 T4321 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2376984 T4278 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4] 2> 2376985 T4278 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 2376986 T4278 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 2377558 T4278 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 2377589 T4278 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52991
[junit4:junit4] 2> 2377595 T4278 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 2377596 T4278 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 2377597 T4278 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371267316985
[junit4:junit4] 2> 2377597 T4278 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371267316985/solr.xml
[junit4:junit4] 2> 2377598 T4278 oasc.CoreContainer.<init> New CoreContainer 653541596
[junit4:junit4] 2> 2377598 T4278 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371267316985/'
[junit4:junit4] 2> 2377599 T4278 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371267316985/'
[junit4:junit4] 2> 2377818 T4278 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 2377820 T4278 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 2377820 T4278 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 2377821 T4278 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 2377821 T4278 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 2377822 T4278 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 2377823 T4278 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 2377823 T4278 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 2377824 T4278 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 2377825 T4278 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 2377878 T4278 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 2377879 T4278 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52981/solr
[junit4:junit4] 2> 2377880 T4278 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 2377894 T4278 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2377900 T4334 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c86ba93 name:ZooKeeperConnection Watcher:127.0.0.1:52981 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2377901 T4278 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2377906 T4278 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 2377961 T4278 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2378048 T4336 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@25463940 name:ZooKeeperConnection Watcher:127.0.0.1:52981/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2378048 T4278 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2378081 T4278 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 2378440 T4301 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2378441 T4301 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52988_m_t",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52988/m_t"}
[junit4:junit4] 2> 2378451 T4336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2378451 T4320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2378453 T4300 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2378460 T4307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2379098 T4278 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52991_m_t
[junit4:junit4] 2> 2379101 T4278 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52991_m_t
[junit4:junit4] 2> 2379106 T4320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2379109 T4336 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 2379110 T4336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2379112 T4300 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 2379113 T4300 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2379113 T4307 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 2379113 T4307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2379129 T4320 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 2379144 T4337 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 2379144 T4337 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 2379967 T4301 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2379971 T4301 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52991_m_t",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52991/m_t"}
[junit4:junit4] 2> 2379972 T4301 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 2379972 T4301 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 2379985 T4320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2379989 T4300 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2379989 T4336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2379991 T4307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2380148 T4337 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371267316985/collection1
[junit4:junit4] 2> 2380149 T4337 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 2380150 T4337 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 2380150 T4337 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 2380154 T4337 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371267316985/collection1/'
[junit4:junit4] 2> 2380155 T4337 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371267316985/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 2380156 T4337 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371267316985/collection1/lib/README' to classloader
[junit4:junit4] 2> 2380278 T4337 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 2380429 T4337 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 2380432 T4337 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 2380442 T4337 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 2381660 T4337 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 2381674 T4337 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 2381683 T4337 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 2381726 T4337 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 2381736 T4337 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 2381744 T4337 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 2381747 T4337 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 2381748 T4337 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 2381748 T4337 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 2381754 T4337 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 2381757 T4337 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 2381758 T4337 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 2381759 T4337 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371267316985/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty2/
[junit4:junit4] 2> 2381759 T4337 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7a8eae04
[junit4:junit4] 2> 2381760 T4337 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 2381761 T4337 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty2
[junit4:junit4] 2> 2381762 T4337 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty2/index/
[junit4:junit4] 2> 2381763 T4337 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 2381764 T4337 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty2/index
[junit4:junit4] 2> 2381772 T4337 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 2381773 T4337 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 2381779 T4337 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 2381780 T4337 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 2381781 T4337 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 2381782 T4337 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 2381795 T4337 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 2381796 T4337 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 2381797 T4337 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 2381798 T4337 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 2381798 T4337 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 2381811 T4337 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 2381824 T4337 oass.SolrIndexSearcher.<init> Opening Searcher@b9578d3 main
[junit4:junit4] 2> 2381826 T4337 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 2381826 T4337 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 2381840 T4338 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@b9578d3 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 2381844 T4337 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 2381844 T4337 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52991/m_t collection:collection1 shard:shard2
[junit4:junit4] 2> 2381846 T4337 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 2381864 T4337 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 2381871 T4337 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 2381871 T4337 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 2381872 T4337 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52991/m_t/collection1/
[junit4:junit4] 2> 2381872 T4337 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 2381872 T4337 oasc.SyncStrategy.syncToMe http://127.0.0.1:52991/m_t/collection1/ has no replicas
[junit4:junit4] 2> 2381873 T4337 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52991/m_t/collection1/
[junit4:junit4] 2> 2381873 T4337 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 2383027 T4301 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2383051 T4336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2383051 T4320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2383055 T4307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2383058 T4300 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2383099 T4337 oasc.ZkController.register We are http://127.0.0.1:52991/m_t/collection1/ and leader is http://127.0.0.1:52991/m_t/collection1/
[junit4:junit4] 2> 2383100 T4337 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52991/m_t
[junit4:junit4] 2> 2383101 T4337 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 2383101 T4337 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 2383102 T4337 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 2383108 T4337 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2383120 T4278 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4] 2> 2383121 T4278 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 2383122 T4278 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 2383760 T4278 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 2383783 T4278 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52994
[junit4:junit4] 2> 2383789 T4278 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 2383790 T4278 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 2383790 T4278 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371267323127
[junit4:junit4] 2> 2383791 T4278 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371267323127/solr.xml
[junit4:junit4] 2> 2383791 T4278 oasc.CoreContainer.<init> New CoreContainer 1444224014
[junit4:junit4] 2> 2383792 T4278 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371267323127/'
[junit4:junit4] 2> 2383793 T4278 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371267323127/'
[junit4:junit4] 2> 2384003 T4278 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 2384004 T4278 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 2384004 T4278 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 2384005 T4278 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 2384005 T4278 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 2384006 T4278 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 2384006 T4278 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 2384007 T4278 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 2384008 T4278 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 2384008 T4278 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 2384060 T4278 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 2384061 T4278 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52981/solr
[junit4:junit4] 2> 2384062 T4278 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 2384064 T4278 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2384069 T4350 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4628950a name:ZooKeeperConnection Watcher:127.0.0.1:52981 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2384071 T4278 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2384076 T4278 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 2384125 T4278 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2384131 T4352 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e849cec name:ZooKeeperConnection Watcher:127.0.0.1:52981/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2384132 T4278 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2384142 T4278 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 2384582 T4301 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2384583 T4301 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52991_m_t",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52991/m_t"}
[junit4:junit4] 2> 2384598 T4352 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2384598 T4336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2384598 T4307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2384599 T4320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2384599 T4300 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2385155 T4278 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52994_m_t
[junit4:junit4] 2> 2385159 T4278 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52994_m_t
[junit4:junit4] 2> 2385165 T4320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2385167 T4300 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 2385167 T4300 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 2385168 T4352 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 2385168 T4307 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 2385168 T4307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 2385168 T4352 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 2385169 T4336 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 2385170 T4336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 2385175 T4320 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 2385187 T4353 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 2385188 T4353 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 2386120 T4301 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2386125 T4301 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52994_m_t",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52994/m_t"}
[junit4:junit4] 2> 2386126 T4301 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 2386127 T4301 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 2386142 T4352 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 2386142 T4300 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 2386145 T4336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 2386145 T4320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 2386146 T4307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 2386192 T4353 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371267323127/collection1
[junit4:junit4] 2> 2386192 T4353 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 2386194 T4353 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 2386194 T4353 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 2386198 T4353 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371267323127/collection1/'
[junit4:junit4] 2> 2386200 T4353 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371267323127/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 2386200 T4353 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371267323127/collection1/lib/README' to classloader
[junit4:junit4] 2> 2386320 T4353 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 2386444 T4353 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 2386447 T4353 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 2386470 T4353 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 2387701 T4353 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 2387715 T4353 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 2387720 T4353 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 2387762 T4353 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 2387772 T4353 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 2387779 T4353 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 2387782 T4353 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 2387783 T4353 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 2387784 T4353 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 2387787 T4353 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 2387787 T4353 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 2387788 T4353 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 2387788 T4353 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371267323127/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty3/
[junit4:junit4] 2> 2387789 T4353 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7a8eae04
[junit4:junit4] 2> 2387789 T4353 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 2387791 T4353 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty3
[junit4:junit4] 2> 2387792 T4353 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty3/index/
[junit4:junit4] 2> 2387793 T4353 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 2387794 T4353 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty3/index
[junit4:junit4] 2> 2387801 T4353 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 2387801 T4353 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 2387806 T4353 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 2387807 T4353 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 2387809 T4353 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 2387810 T4353 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 2387811 T4353 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 2387811 T4353 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 2387812 T4353 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 2387813 T4353 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 2387817 T4353 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 2387832 T4353 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 2387845 T4353 oass.SolrIndexSearcher.<init> Opening Searcher@4342bd6e main
[junit4:junit4] 2> 2387847 T4353 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 2387847 T4353 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 2387879 T4354 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4342bd6e main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 2387891 T4353 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 2387892 T4353 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52994/m_t collection:collection1 shard:shard3
[junit4:junit4] 2> 2387893 T4353 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4:junit4] 2> 2387911 T4353 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 2387920 T4353 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 2387920 T4353 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 2387921 T4353 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52994/m_t/collection1/
[junit4:junit4] 2> 2387921 T4353 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 2387922 T4353 oasc.SyncStrategy.syncToMe http://127.0.0.1:52994/m_t/collection1/ has no replicas
[junit4:junit4] 2> 2387922 T4353 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52994/m_t/collection1/
[junit4:junit4] 2> 2387923 T4353 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4:junit4] 2> 2389210 T4301 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2389244 T4352 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 2389245 T4300 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 2389249 T4307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 2389250 T4336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 2389250 T4320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 2389285 T4353 oasc.ZkController.register We are http://127.0.0.1:52994/m_t/collection1/ and leader is http://127.0.0.1:52994/m_t/collection1/
[junit4:junit4] 2> 2389286 T4353 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52994/m_t
[junit4:junit4] 2> 2389286 T4353 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 2389286 T4353 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 2389287 T4353 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 2389290 T4353 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2389293 T4278 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4] 2> 2389294 T4278 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 2389294 T4278 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 2389855 T4278 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 2389861 T4278 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52997
[junit4:junit4] 2> 2389864 T4278 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 2389865 T4278 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 2389866 T4278 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371267329298
[junit4:junit4] 2> 2389866 T4278 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371267329298/solr.xml
[junit4:junit4] 2> 2389867 T4278 oasc.CoreContainer.<init> New CoreContainer 806182611
[junit4:junit4] 2> 2389868 T4278 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371267329298/'
[junit4:junit4] 2> 2389869 T4278 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371267329298/'
[junit4:junit4] 2> 2390078 T4278 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 2390079 T4278 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 2390079 T4278 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 2390080 T4278 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 2390080 T4278 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 2390081 T4278 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 2390081 T4278 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 2390082 T4278 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 2390083 T4278 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 2390083 T4278 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 2390134 T4278 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 2390135 T4278 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52981/solr
[junit4:junit4] 2> 2390136 T4278 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 2390137 T4278 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2390143 T4366 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@418fc088 name:ZooKeeperConnection Watcher:127.0.0.1:52981 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2390145 T4278 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2390149 T4278 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 2390149 T4280 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x13f45e68d9e000b, likely client has closed socket
[junit4:junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
[junit4:junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:680)
[junit4:junit4] 2>
[junit4:junit4] 2> 2390196 T4278 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2390202 T4368 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@43a6b049 name:ZooKeeperConnection Watcher:127.0.0.1:52981/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2390203 T4278 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2390213 T4278 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 2390768 T4301 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2390770 T4301 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52994_m_t",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52994/m_t"}
[junit4:junit4] 2> 2390781 T4300 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 2390781 T4320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 2390783 T4307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 2390791 T4352 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 2390791 T4368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 2390791 T4336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 2391226 T4278 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52997_m_t
[junit4:junit4] 2> 2391230 T4278 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52997_m_t
[junit4:junit4] 2> 2391236 T4320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 2391239 T4336 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 2391239 T4336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 2391239 T4307 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 2391240 T4307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 2391241 T4368 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 2391241 T4368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 2391242 T4300 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 2391242 T4300 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 2391242 T4352 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 2391242 T4352 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 2391254 T4320 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 2391278 T4369 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 2391279 T4369 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 2392307 T4301 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2392310 T4301 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52997_m_t",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52997/m_t"}
[junit4:junit4] 2> 2392310 T4301 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 2392310 T4301 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 2392322 T4352 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 2392323 T4300 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 2392323 T4368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 2392329 T4307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 2392332 T4336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 2392336 T4320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 2393289 T4369 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371267329298/collection1
[junit4:junit4] 2> 2393289 T4369 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 2393295 T4369 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 2393298 T4369 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 2393304 T4369 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371267329298/collection1/'
[junit4:junit4] 2> 2393306 T4369 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371267329298/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 2393311 T4369 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371267329298/collection1/lib/README' to classloader
[junit4:junit4] 2> 2393482 T4369 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 2393650 T4369 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 2393656 T4369 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 2393665 T4369 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 2395020 T4369 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 2395035 T4369 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 2395042 T4369 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 2395083 T4369 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 2395094 T4369 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 2395103 T4369 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 2395107 T4369 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 2395108 T4369 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 2395109 T4369 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 2395112 T4369 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 2395113 T4369 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 2395113 T4369 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 2395114 T4369 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371267329298/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty4/
[junit4:junit4] 2> 2395114 T4369 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7a8eae04
[junit4:junit4] 2> 2395115 T4369 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 2395116 T4369 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty4
[junit4:junit4] 2> 2395117 T4369 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty4/index/
[junit4:junit4] 2> 2395118 T4369 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 2395119 T4369 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty4/index
[junit4:junit4] 2> 2395130 T4369 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 2395134 T4369 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 2395140 T4369 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 2395141 T4369 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 2395142 T4369 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 2395143 T4369 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 2395144 T4369 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 2395144 T4369 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 2395145 T4369 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 2395146 T4369 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 2395149 T4369 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 2395161 T4369 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 2395174 T4369 oass.SolrIndexSearcher.<init> Opening Searcher@2d60226a main
[junit4:junit4] 2> 2395177 T4369 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 2395177 T4369 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 2395207 T4371 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2d60226a main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 2395211 T4369 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 2395212 T4369 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52997/m_t collection:collection1 shard:shard1
[junit4:junit4] 2> 2395221 T4369 oasc.ZkController.register We are http://127.0.0.1:52997/m_t/collection1/ and leader is http://127.0.0.1:52988/m_t/collection1/
[junit4:junit4] 2> 2395222 T4369 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52997/m_t
[junit4:junit4] 2> 2395222 T4369 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 2395222 T4369 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C3171 name=collection1 org.apache.solr.core.SolrCore@29ac019f url=http://127.0.0.1:52997/m_t/collection1 node=127.0.0.1:52997_m_t C3171_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:52997_m_t, base_url=http://127.0.0.1:52997/m_t}
[junit4:junit4] 2> 2395224 T4372 C3171 P52997 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 2395225 T4372 C3171 P52997 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 2395226 T4372 C3171 P52997 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 2395226 T4372 C3171 P52997 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 2395228 T4369 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2395239 T4372 C3171 P52997 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 2395250 T4278 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4] 2> 2395251 T4278 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 2395252 T4278 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 2395330 T4316 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 2395393 T4301 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2395395 T4301 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52997_m_t",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52997/m_t"}
[junit4:junit4] 2> 2395413 T4307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 2395413 T4336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 2395418 T4320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 2395414 T4368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 2395414 T4352 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 2395414 T4300 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 2395929 T4278 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 2395979 T4278 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53002
[junit4:junit4] 2> 2395985 T4278 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 2395986 T4278 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 2395987 T4278 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371267335277
[junit4:junit4] 2> 2395987 T4278 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371267335277/solr.xml
[junit4:junit4] 2> 2395988 T4278 oasc.CoreContainer.<init> New CoreContainer 2127688689
[junit4:junit4] 2> 2395989 T4278 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371267335277/'
[junit4:junit4] 2> 2395990 T4278 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371267335277/'
[junit4:junit4] 2> 2396195 T4278 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 2396197 T4278 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 2396198 T4278 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 2396199 T4278 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 2396200 T4278 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 2396201 T4278 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 2396201 T4278 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 2396202 T4278 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 2396203 T4278 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 2396203 T4278 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 2396256 T4278 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 2396257 T4278 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52981/solr
[junit4:junit4] 2> 2396258 T4278 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 2396269 T4278 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2396276 T4384 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7370d503 name:ZooKeeperConnection Watcher:127.0.0.1:52981 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2396278 T4278 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2396289 T4278 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 2396332 T4316 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 2396333 T4316 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=5&state=recovering&nodeName=127.0.0.1:52997_m_t&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1003
[junit4:junit4] 2> 2396341 T4278 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2396347 T4386 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6fe76ebb name:ZooKeeperConnection Watcher:127.0.0.1:52981/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2396347 T4278 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2396360 T4278 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 2397373 T4278 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53002_m_t
[junit4:junit4] 2> 2397381 T4278 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53002_m_t
[junit4:junit4] 2> 2397395 T4368 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 2397395 T4368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 2397396 T4386 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 2397396 T4300 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 2397397 T4300 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 2397397 T4307 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 2397398 T4307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 2397408 T4336 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 2397409 T4336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 2397414 T4320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 2397420 T4320 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 2397422 T4352 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 2397422 T4352 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 2397449 T4387 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 2397450 T4387 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> ASYNC NEW_CORE C3172 name=collection1 org.apache.solr.core.SolrCore@29ac019f url=http://127.0.0.1:52997/m_t/collection1 node=127.0.0.1:52997_m_t C3172_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:52997_m_t, base_url=http://127.0.0.1:52997/m_t}
[junit4:junit4] 2> 2398344 T4372 C3172 P52997 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:52988/m_t/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 2398345 T4372 C3172 P52997 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:52997/m_t START replicas=[http://127.0.0.1:52988/m_t/collection1/] nUpdates=100
[junit4:junit4] 2> 2398345 T4372 C3172 P52997 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 2398346 T4372 C3172 P52997 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 2398346 T4372 C3172 P52997 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 2398346 T4372 C3172 P52997 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 2398346 T4372 C3172 P52997 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 2398347 T4372 C3172 P52997 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:52988/m_t/collection1/. core=collection1
[junit4:junit4] 2> 2398347 T4372 C3172 P52997 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C3173 name=collection1 org.apache.solr.core.SolrCore@55f677e2 url=http://127.0.0.1:52988/m_t/collection1 node=127.0.0.1:52988_m_t C3173_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:52988_m_t, base_url=http://127.0.0.1:52988/m_t, leader=true}
[junit4:junit4] 2> 2398389 T4313 C3173 P52988 oasc.SolrCore.execute [collection1] webapp=/m_t path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=3
[junit4:junit4] 2> 2398414 T4315 C3173 P52988 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 2398424 T4315 C3173 P52988 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 2398428 T4315 C3173 P52988 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 2398438 T4315 C3173 P52988 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 2398439 T4315 C3173 P52988 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 2398442 T4315 C3173 P52988 oass.SolrIndexSearcher.<init> Opening Searcher@5bf40589 realtime
[junit4:junit4] 2> 2398443 T4315 C3173 P52988 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 2398444 T4315 C3173 P52988 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_t path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 30
[junit4:junit4] 2> 2398445 T4301 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2398446 T4372 C3172 P52997 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 2398446 T4372 C3172 P52997 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 2398456 T4301 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53002_m_t",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53002/m_t"}
[junit4:junit4] 2> 2398457 T4301 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 2398457 T4301 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 2398459 T4312 C3173 P52988 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 2398460 T4312 C3173 P52988 oasc.SolrCore.execute [collection1] webapp=/m_t path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2
[junit4:junit4] 2> 2398461 T4372 C3172 P52997 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 2398462 T4372 C3172 P52997 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 2398462 T4372 C3172 P52997 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 2398466 T4316 C3173 P52988 oasc.SolrCore.execute [collection1] webapp=/m_t path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=1
[junit4:junit4] 2> 2398468 T4372 C3172 P52997 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 2398472 T4372 C3172 P52997 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty4/index.20130615003538420
[junit4:junit4] 2> 2398474 T4372 C3172 P52997 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.NIOFSDirectory@/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty4/index.20130615003538420 lockFactory=org.apache.lucene.store.NativeFSLockFactory@46f71800 fullCopy=false
[junit4:junit4] 2> 2398487 T4336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 2398488 T4368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 2398488 T4352 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 2398489 T4307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 2398490 T4320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 2398491 T4313 C3173 P52988 oasc.SolrCore.execute [collection1] webapp=/m_t path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1
[junit4:junit4] 2> 2398492 T4300 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 2398499 T4386 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 2398519 T4372 C3172 P52997 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 2398523 T4372 C3172 P52997 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 2398523 T4372 C3172 P52997 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 2398539 T4372 C3172 P52997 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty4/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 2398540 T4372 C3172 P52997 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 2398541 T4372 C3172 P52997 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 2398543 T4372 C3172 P52997 oass.SolrIndexSearcher.<init> Opening Searcher@69710514 main
[junit4:junit4] 2> 2398544 T4371 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@69710514 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 2398545 T4372 C3172 P52997 oasc.CachingDirectoryFactory.closeCacheValue looking to close /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty4/index.20130615003538420 [CachedDir<<refCount=0;path=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty4/index.20130615003538420;done=true>>]
[junit4:junit4] 2> 2398545 T4372 C3172 P52997 oasc.CachingDirectoryFactory.close Closing directory: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty4/index.20130615003538420
[junit4:junit4] 2> 2398546 T4372 C3172 P52997 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty4/index.20130615003538420
[junit4:junit4] 2> 2398547 T4372 C3172 P52997 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> ASYNC NEW_CORE C3174 name=collection1 org.apache.solr.core.SolrCore@29ac019f url=http://127.0.0.1:52997/m_t/collection1 node=127.0.0.1:52997_m_t C3174_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:52997_m_t, base_url=http://127.0.0.1:52997/m_t}
[junit4:junit4] 2> 2398561 T4372 C3174 P52997 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 2398561 T4372 C3174 P52997 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 2398562 T4372 C3174 P52997 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 2398564 T4372 C3174 P52997 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 2399455 T4387 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371267335277/collection1
[junit4:junit4] 2> 2399456 T4387 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 2399457 T4387 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 2399458 T4387 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 2399461 T4387 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371267335277/collection1/'
[junit4:junit4] 2> 2399462 T4387 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371267335277/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 2399463 T4387 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371267335277/collection1/lib/README' to classloader
[junit4:junit4] 2> 2399615 T4387 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 2399754 T4387 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 2399758 T4387 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 2399766 T4387 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 2400120 T4301 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2400122 T4301 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52997_m_t",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52997/m_t"}
[junit4:junit4] 2> 2400133 T4336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 2400133 T4368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 2400133 T4352 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 2400133 T4307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 2400133 T4386 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 2400133 T4320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 2400134 T4300 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 2401042 T4387 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 2401058 T4387 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 2401063 T4387 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 2401120 T4387 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 2401131 T4387 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 2401139 T4387 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 2401142 T4387 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 2401143 T4387 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 2401144 T4387 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 2401148 T4387 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 2401149 T4387 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 2401150 T4387 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 2401151 T4387 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371267335277/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty5/
[junit4:junit4] 2> 2401152 T4387 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7a8eae04
[junit4:junit4] 2> 2401154 T4387 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 2401158 T4387 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty5
[junit4:junit4] 2> 2401160 T4387 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty5/index/
[junit4:junit4] 2> 2401161 T4387 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty5/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 2401163 T4387 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty5/index
[junit4:junit4] 2> 2401172 T4387 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 2401172 T4387 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 2401179 T4387 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 2401179 T4387 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 2401180 T4387 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 2401182 T4387 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 2401183 T4387 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 2401183 T4387 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 2401188 T4387 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 2401189 T4387 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 2401189 T4387 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 2401202 T4387 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 2401219 T4387 oass.SolrIndexSearcher.<init> Opening Searcher@464095c2 main
[junit4:junit4] 2> 2401221 T4387 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 2401222 T4387 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 2401231 T4390 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@464095c2 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 2401234 T4387 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 2401234 T4387 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53002/m_t collection:collection1 shard:shard2
[junit4:junit4] 2> 2401242 T4387 oasc.ZkController.register We are http://127.0.0.1:53002/m_t/collection1/ and leader is http://127.0.0.1:52991/m_t/collection1/
[junit4:junit4] 2> 2401243 T4387 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53002/m_t
[junit4:junit4] 2> 2401243 T4387 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 2401243 T4387 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C3175 name=collection1 org.apache.solr.core.SolrCore@14f23fd3 url=http://127.0.0.1:53002/m_t/collection1 node=127.0.0.1:53002_m_t C3175_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:53002_m_t, base_url=http://127.0.0.1:53002/m_t}
[junit4:junit4] 2> 2401245 T4391 C3175 P53002 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 2401247 T4391 C3175 P53002 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 2401247 T4391 C3175 P53002 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 2401247 T4391 C3175 P53002 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 2401249 T4391 C3175 P53002 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 2401260 T4387 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2401296 T4330 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 6, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 2401301 T4278 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4] 2> 2401306 T4278 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 2401307 T4278 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 2401697 T4301 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2401703 T4301 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53002_m_t",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53002/m_t"}
[junit4:junit4] 2> 2401729 T4307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 2401730 T4300 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 2401729 T4336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 2401734 T4386 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 2401729 T4320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 2401731 T4352 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 2401731 T4368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 2401956 T4278 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 2401965 T4278 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53009
[junit4:junit4] 2> 2401968 T4278 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 2401969 T4278 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 2401970 T4278 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371267341302
[junit4:junit4] 2> 2401971 T4278 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371267341302/solr.xml
[junit4:junit4] 2> 2401971 T4278 oasc.CoreContainer.<init> New CoreContainer 2086580606
[junit4:junit4] 2> 2401972 T4278 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371267341302/'
[junit4:junit4] 2> 2401972 T4278 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371267341302/'
[junit4:junit4] 2> 2402190 T4278 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 2402191 T4278 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 2402191 T4278 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 2402192 T4278 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 2402192 T4278 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 2402193 T4278 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 2402194 T4278 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 2402194 T4278 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 2402195 T4278 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 2402196 T4278 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 2402250 T4278 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 2402252 T4278 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52981/solr
[junit4:junit4] 2> 2402252 T4278 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 2402269 T4278 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2402276 T4403 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a909841 name:ZooKeeperConnection Watcher:127.0.0.1:52981 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2402278 T4278 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2402283 T4278 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 2402298 T4330 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 6, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 2402298 T4330 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=6&state=recovering&nodeName=127.0.0.1:53002_m_t&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1002
[junit4:junit4] 2> 2402339 T4278 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2402345 T4405 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a62c6b7 name:ZooKeeperConnection Watcher:127.0.0.1:52981/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2402345 T4278 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2402362 T4278 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 2403379 T4278 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53009_m_t
[junit4:junit4] 2> 2403382 T4278 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53009_m_t
[junit4:junit4] 2> 2403393 T4320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 2403396 T4352 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 2403396 T4352 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 2403397 T4405 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 2403397 T4307 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 2403398 T4307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 2403406 T4368 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 2403406 T4368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 2403416 T4406 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 2403416 T4406 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 2403419 T4386 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 2403419 T4386 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 2403423 T4300 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 2403423 T4300 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 2403424 T4336 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 2403425 T4336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 2403434 T4320 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> ASYNC NEW_CORE C3176 name=collection1 org.apache.solr.core.SolrCore@14f23fd3 url=http://127.0.0.1:53002/m_t/collection1 node=127.0.0.1:53002_m_t C3176_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:53002_m_t, base_url=http://127.0.0.1:53002/m_t}
[junit4:junit4] 2> 2404304 T4391 C3176 P53002 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:52991/m_t/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 2404304 T4391 C3176 P53002 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:53002/m_t START replicas=[http://127.0.0.1:52991/m_t/collection1/] nUpdates=100
[junit4:junit4] 2> 2404305 T4391 C3176 P53002 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 2404305 T4391 C3176 P53002 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 2404306 T4391 C3176 P53002 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 2404306 T4391 C3176 P53002 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 2404308 T4391 C3176 P53002 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 2404310 T4391 C3176 P53002 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:52991/m_t/collection1/. core=collection1
[junit4:junit4] 2> 2404311 T4391 C3176 P53002 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C3177 name=collection1 org.apache.solr.core.SolrCore@47e99eda url=http://127.0.0.1:52991/m_t/collection1 node=127.0.0.1:52991_m_t C3177_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:52991_m_t, base_url=http://127.0.0.1:52991/m_t, leader=true}
[junit4:junit4] 2> 2404361 T4327 C3177 P52991 oasc.SolrCore.execute [collection1] webapp=/m_t path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=2
[junit4:junit4] 2> 2404393 T4328 C3177 P52991 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 2404399 T4328 C3177 P52991 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 2404401 T4328 C3177 P52991 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 2404406 T4328 C3177 P52991 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty2/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 2404406 T4328 C3177 P52991 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 2404409 T4328 C3177 P52991 oass.SolrIndexSearcher.<init> Opening Searcher@43c767ac realtime
[junit4:junit4] 2> 2404410 T4328 C3177 P52991 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 2404410 T4328 C3177 P52991 oasup.LogUpdateProcessor.finish [collection1] webapp=/m_t path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 18
[junit4:junit4] 2> ASYNC NEW_CORE C3178 name=collection1 org.apache.solr.core.SolrCore@14f23fd3 url=http://127.0.0.1:53002/m_t/collection1 node=127.0.0.1:53002_m_t C3178_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:53002_m_t, base_url=http://127.0.0.1:53002/m_t}
[junit4:junit4] 2> 2404413 T4391 C3178 P53002 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 2404413 T4391 C3178 P53002 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 2404421 T4331 C3177 P52991 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 2404422 T4331 C3177 P52991 oasc.SolrCore.execute [collection1] webapp=/m_t path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2
[junit4:junit4] 2> 2404423 T4391 C3178 P53002 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 2404423 T4391 C3178 P53002 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 2404424 T4391 C3178 P53002 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 2404433 T4325 C3177 P52991 oasc.SolrCore.execute [collection1] webapp=/m_t path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=1
[junit4:junit4] 2> 2404435 T4391 C3178 P53002 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 2404437 T4391 C3178 P53002 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty5/index.20130615003544387
[junit4:junit4] 2> 2404438 T4391 C3178 P53002 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.NIOFSDirectory@/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty5/index.20130615003544387 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5d4d2b69 fullCopy=false
[junit4:junit4] 2> 2404448 T4330 C3177 P52991 oasc.SolrCore.execute [collection1] webapp=/m_t path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1
[junit4:junit4] 2> 2404453 T4391 C3178 P53002 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 2404457 T4391 C3178 P53002 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 2404457 T4391 C3178 P53002 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 2404464 T4391 C3178 P53002 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty5/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 2404465 T4391 C3178 P53002 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 2404465 T4391 C3178 P53002 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 2404467 T4391 C3178 P53002 oass.SolrIndexSearcher.<init> Opening Searcher@700656af main
[junit4:junit4] 2> 2404469 T4390 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@700656af main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 2404470 T4391 C3178 P53002 oasc.CachingDirectoryFactory.closeCacheValue looking to close /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty5/index.20130615003544387 [CachedDir<<refCount=0;path=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty5/index.20130615003544387;done=true>>]
[junit4:junit4] 2> 2404470 T4391 C3178 P53002 oasc.CachingDirectoryFactory.close Closing directory: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty5/index.20130615003544387
[junit4:junit4] 2> 2404471 T4391 C3178 P53002 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty5/index.20130615003544387
[junit4:junit4] 2> 2404472 T4391 C3178 P53002 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 2404472 T4391 C3178 P53002 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 2404472 T4391 C3178 P53002 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 2404473 T4391 C3178 P53002 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 2404475 T4391 C3178 P53002 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 2404773 T4301 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2404776 T4301 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"7",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53009_m_t",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53009/m_t"}
[junit4:junit4] 2> 2404777 T4301 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 2404778 T4301 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 2404787 T4301 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53002_m_t",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53002/m_t"}
[junit4:junit4] 2> 2404799 T4320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 2404799 T4386 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 2404799 T4300 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 2404813 T4336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 2404813 T4368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 2404814 T4352 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 2404817 T4307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 2404821 T4405 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 2405425 T4406 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371267341302/collection1
[junit4:junit4] 2> 2405426 T4406 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 2405427 T4406 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 2405428 T4406 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 2405431 T4406 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371267341302/collection1/'
[junit4:junit4] 2> 2405433 T4406 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371267341302/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 2405434 T4406 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371267341302/collection1/lib/README' to classloader
[junit4:junit4] 2> 2405646 T4406 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 2405796 T4406 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 2405799 T4406 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 2405807 T4406 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 2407130 T4406 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 2407147 T4406 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 2407153 T4406 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 2407197 T4406 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 2407206 T4406 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 2407214 T4406 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 2407217 T4406 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 2407219 T4406 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 2407219 T4406 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 2407222 T4406 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 2407223 T4406 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 2407223 T4406 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 2407224 T4406 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371267341302/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty6/
[junit4:junit4] 2> 2407224 T4406 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7a8eae04
[junit4:junit4] 2> 2407225 T4406 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 2407226 T4406 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty6
[junit4:junit4] 2> 2407227 T4406 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty6/index/
[junit4:junit4] 2> 2407228 T4406 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty6/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 2407229 T4406 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty6/index
[junit4:junit4] 2> 2407238 T4406 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty6/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 2407238 T4406 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 2407244 T4406 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 2407245 T4406 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 2407247 T4406 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 2407248 T4406 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 2407249 T4406 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 2407250 T4406 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 2407257 T4406 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 2407261 T4406 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 2407262 T4406 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 2407276 T4406 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 2407289 T4406 oass.SolrIndexSearcher.<init> Opening Searcher@2dffd830 main
[junit4:junit4] 2> 2407291 T4406 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 2407291 T4406 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 2407298 T4409 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2dffd830 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 2407303 T4406 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 2407303 T4406 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53009/m_t collection:collection1 shard:shard3
[junit4:junit4] 2> 2407328 T4406 oasc.ZkController.register We are http://127.0.0.1:53009/m_t/collection1/ and leader is http://127.0.0.1:52994/m_t/collection1/
[junit4:junit4] 2> 2407331 T4406 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53009/m_t
[junit4:junit4] 2> 2407335 T4406 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 2407336 T4406 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C3179 name=collection1 org.apache.solr.core.SolrCore@2d415de url=http://127.0.0.1:53009/m_t/collection1 node=127.0.0.1:53009_m_t C3179_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:53009_m_t, base_url=http://127.0.0.1:53009/m_t}
[junit4:junit4] 2> 2407337 T4410 C3179 P53009 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 2407342 T4410 C3179 P53009 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 2407342 T4410 C3179 P53009 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 2407343 T4406 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2407343 T4410 C3179 P53009 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 2407346 T4410 C3179 P53009 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 2407364 T4278 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4] 2> 2407365 T4278 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 2407366 T4278 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 2407450 T4345 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 7, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 2407875 T4301 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2407880 T4301 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"7",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53009_m_t",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53009/m_t"}
[junit4:junit4] 2> 2407893 T4307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 2407893 T4320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 2407894 T4386 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 2407894 T4352 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 2407902 T4368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 2407903 T4300 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 2407904 T4336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 2407904 T4405 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 2408076 T4278 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 2408102 T4278 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53016
[junit4:junit4] 2> 2408121 T4278 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 2408123 T4278 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 2408123 T4278 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371267347380
[junit4:junit4] 2> 2408124 T4278 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371267347380/solr.xml
[junit4:junit4] 2> 2408125 T4278 oasc.CoreContainer.<init> New CoreContainer 103954571
[junit4:junit4] 2> 2408126 T4278 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371267347380/'
[junit4:junit4] 2> 2408127 T4278 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371267347380/'
[junit4:junit4] 2> 2408355 T4278 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 2408356 T4278 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 2408356 T4278 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 2408357 T4278 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 2408358 T4278 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 2408359 T4278 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 2408359 T4278 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 2408360 T4278 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 2408361 T4278 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 2408361 T4278 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 2408414 T4278 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 2408416 T4278 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52981/solr
[junit4:junit4] 2> 2408417 T4278 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 2408420 T4278 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2408426 T4422 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a025007 name:ZooKeeperConnection Watcher:127.0.0.1:52981 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2408427 T4278 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2408432 T4278 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 2408452 T4345 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 7, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 2408453 T4345 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=7&state=recovering&nodeName=127.0.0.1:53009_m_t&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1003
[junit4:junit4] 2> 2408491 T4278 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2408499 T4424 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@915ca3a name:ZooKeeperConnection Watcher:127.0.0.1:52981/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2408500 T4278 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2408520 T4278 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 2409535 T4278 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53016_m_t
[junit4:junit4] 2> 2409539 T4278 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53016_m_t
[junit4:junit4] 2> 2409545 T4405 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 2409546 T4320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 2409551 T4386 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 2409551 T4386 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 2409553 T4368 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 2409553 T4368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 2409561 T4405 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 2409563 T4424 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 2409587 T4425 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 2409588 T4307 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 2409588 T4307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 2409589 T4320 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 2409588 T4425 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 2409590 T4352 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 2409590 T4336 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 2409591 T4336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 2409591 T4352 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 2409590 T4300 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 2409593 T4300 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> ASYNC NEW_CORE C3180 name=collection1 org.apache.solr.core.SolrCore@2d415de url=http://127.0.0.1:53009/m_t/collection1 node=127.0.0.1:53009_m_t C3180_STATE=coll:collection1 core:collection1 props:{shard=shard3, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:53009_m_t, base_url=http://127.0.0.1:53009/m_t}
[junit4:junit4] 2> 2410456 T4410 C3180 P53009 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:52994/m_t/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 2410457 T4410 C3180 P53009 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:53009/m_t START replicas=[http://127.0.0.1:52994/m_t/collection1/] nUpdates=100
[junit4:junit4] 2> 2410457 T4410 C3180 P53009 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 2410458 T4410 C3180 P53009 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 2410458 T4410 C3180 P53009 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 2410458 T4410 C3180 P53009 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 2410459 T4410 C3180 P53009 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 2410459 T4410 C3180 P53009 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:52994/m_t/collection1/. core=collection1
[junit4:junit4] 2> 2410459 T4410 C3180 P53009 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C3181 name=collection1 org.apache.solr.core.SolrCore@229b56d5 url=http://127.0.0.1:52994/m_t/collection1 node=127.0.0.1:52994_m_t C3181_STATE=coll:collection1 core:collection1 props:{shard=shard3, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:52994_m_t, base_url=http://127.0.0.1:52994/m_t, leader=true}
[junit4:junit4] 2> 2410511 T4348 C3181 P52994 oasc.Sol
[...truncated too long message...]
riterCloser
[junit4:junit4] 2> 2524200 T4278 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 2524202 T4278 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 2524202 T4278 oasc.CachingDirectoryFactory.closeCacheValue looking to close /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty12/index [CachedDir<<refCount=0;path=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty12/index;done=false>>]
[junit4:junit4] 2> 2524203 T4278 oasc.CachingDirectoryFactory.close Closing directory: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty12/index
[junit4:junit4] 2> 2524203 T4278 oasc.CachingDirectoryFactory.closeCacheValue looking to close /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty12 [CachedDir<<refCount=0;path=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty12;done=false>>]
[junit4:junit4] 2> 2524204 T4278 oasc.CachingDirectoryFactory.close Closing directory: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371267304848/jetty12
[junit4:junit4] 2> 2524205 T4642 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89867374091436060-127.0.0.1:53043_m_t-n_0000000012) am no longer a leader.
[junit4:junit4] 2> 2524233 T4278 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/m_t,null}
[junit4:junit4] 2> 2524398 T4579 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 2524398 T4579 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 2524449 T4278 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 2524459 T4278 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:52981 52981
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.method=testDistribSearch -Dtests.seed=7C6E80877E7D058 -Dtests.slow=true -Dtests.locale=en_US -Dtests.timezone=Brazil/East -Dtests.file.encoding=US-ASCII
[junit4:junit4] FAILURE 160s | ChaosMonkeySafeLeaderTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: shard1 is not consistent. Got 119 from http://127.0.0.1:52988/m_t/collection1lastClient and got 80 from http://127.0.0.1:53016/m_t/collection1
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([7C6E80877E7D058:8620661000B8B064]:0)
[junit4:junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:963)
[junit4:junit4] > at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:137)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:680)
[junit4:junit4] 2> 2524678 T4278 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 159824 T4277 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4] 2> 2525518 T4519 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 2525519 T4519 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 2525519 T4519 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {range_facet_l=PostingsFormat(name=Lucene41WithOrds), text=PostingsFormat(name=TestBloomFilteredLucene41Postings), _version_=PostingsFormat(name=Lucene41WithOrds), rnd_b=Pulsing41(freqCutoff=11 minBlockSize=20 maxBlockSize=113), intDefault=PostingsFormat(name=Lucene41WithOrds), id=Pulsing41(freqCutoff=11 minBlockSize=20 maxBlockSize=113), timestamp=PostingsFormat(name=Lucene41WithOrds), a_t=PostingsFormat(name=SimpleText), range_facet_sl=Pulsing41(freqCutoff=11 minBlockSize=20 maxBlockSize=113), range_facet_si=PostingsFormat(name=SimpleText), other_tl1=PostingsFormat(name=Lucene41WithOrds), multiDefault=PostingsFormat(name=Lucene41WithOrds), a_si=PostingsFormat(name=Lucene41WithOrds)}, docValues:{}, sim=DefaultSimilarity, locale=en_US, timezone=Brazil/East
[junit4:junit4] 2> NOTE: Mac OS X 10.8.4 x86_64/Apple Inc. 1.6.0_45 (64-bit)/cpus=2,threads=1,free=45287288,total=480739328
[junit4:junit4] 2> NOTE: All tests run in this JVM: [CurrencyFieldOpenExchangeTest, StandardRequestHandlerTest, ZkControllerTest, TestHashPartitioner, DocValuesTest, TestCSVLoader, FullSolrCloudDistribCmdsTest, TestUniqueKeyFieldResource, QueryParsingTest, ShowFileRequestHandlerTest, CSVRequestHandlerTest, OverseerCollectionProcessorTest, AutoCommitTest, ConvertedLegacyTest, SuggesterTSTTest, FieldAnalysisRequestHandlerTest, TestStressLucene, TestLFUCache, TestRemoteStreaming, TestArbitraryIndexDir, PreAnalyzedUpdateProcessorTest, OverseerTest, TestDistributedSearch, TestFaceting, RequestHandlersTest, SuggesterTest, BadIndexSchemaTest, DirectSolrSpellCheckerTest, ExternalFileFieldSortTest, SolrIndexConfigTest, TestGroupingSearch, DirectSolrConnectionTest, TestFuzzyAnalyzedSuggestions, CachingDirectoryFactoryTest, TestSurroundQueryParser, TestDynamicFieldCollectionResource, TestFieldResource, PolyFieldTest, TestReloadAndDeleteDocs, PreAnalyzedFieldTest, TestFastWriter, TestPropInjectDefaults, TestFieldCollectionResource, TestSchemaSimilarityResource, TestAnalyzedSuggestions, TestJmxMonitoredMap, SolrCmdDistributorTest, TestNumberUtils, DateMathParserTest, DocumentBuilderTest, TestTrie, IndexSchemaTest, TestReplicationHandler, SpellPossibilityIteratorTest, TestQueryUtils, FileBasedSpellCheckerTest, TestElisionMultitermQuery, URLClassifyProcessorTest, TestPseudoReturnFields, TestCloudManagedSchemaAddField, TestQuerySenderListener, BasicZkTest, BasicFunctionalityTest, SynonymTokenizerTest, ScriptEngineTest, PathHierarchyTokenizerFactoryTest, FastVectorHighlighterTest, ShardSplitTest, QueryResultKeyTest, TestMergePolicyConfig, OutputWriterTest, FileUtilsTest, TestQueryTypes, QueryElevationComponentTest, TestFiltering, DefaultValueUpdateProcessorTest, TestSweetSpotSimilarityFactory, SpatialFilterTest, TestCoreDiscovery, TestBinaryField, TestReversedWildcardFilterFactory, DirectUpdateHandlerOptimizeTest, HighlighterConfigTest, PingRequestHandlerTest, TestRecovery, ResponseLogComponentTest, TestDefaultSimilarityFactory, SchemaVersionSpecificBehaviorTest, MultiTermTest, TestWordDelimiterFilterFactory, RegexBoostProcessorTest, PrimUtilsTest, DateFieldTest, TestPostingsSolrHighlighter, TestIBSimilarityFactory, TestShardHandlerFactory, TestOmitPositions, BinaryUpdateRequestHandlerTest, ShardRoutingCustomTest, TestSolrQueryParserDefaultOperatorResource, ZkCLITest, ClusterStateTest, TestConfig, TestMultiCoreConfBootstrap, JSONWriterTest, UnloadDistributedZkTest, SolrRequestParserTest, LeaderElectionIntegrationTest, TimeZoneUtilsTest, TestUpdate, CacheHeaderTest, MBeansHandlerTest, XsltUpdateRequestHandlerTest, ClusterStateUpdateTest, DirectUpdateHandlerTest, TestLMJelinekMercerSimilarityFactory, ZkNodePropsTest, TestFastOutputStream, DistributedSpellCheckComponentTest, TestFieldTypeResource, SolrCoreTest, TestCopyFieldCollectionResource, SyncSliceTest, TestRangeQuery, LukeRequestHandlerTest, ResourceLoaderTest, TestAddFieldRealTimeGet, AnalysisAfterCoreReloadTest, StatsComponentTest, DebugComponentTest, CollectionsAPIDistributedZkTest, TestCoreContainer, LeaderElectionTest, TestValueSourceCache, TestSystemIdResolver, NoCacheHeaderTest, TestRandomFaceting, TestAtomicUpdateErrorCases, HighlighterTest, TestClassNameShortening, TestRTGBase, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest]
[junit4:junit4] Completed in 160.92s, 1 test, 1 failure <<< FAILURES!
[...truncated 465 lines...]
BUILD FAILED
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/build.xml:391: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/build.xml:371: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build.xml:181: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/common-build.xml:437: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:1243: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:887: There were test failures: 297 suites, 1246 tests, 1 failure, 18 ignored (6 assumptions)
Total time: 121 minutes 50 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.6.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-4.x-MacOSX (64bit/jdk1.6.0) - Build # 549 -
Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/549/
Java: 64bit/jdk1.6.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
1 tests failed.
REGRESSION: org.apache.solr.cloud.ShardSplitTest.testDistribSearch
Error Message:
Server at http://127.0.0.1:53873/eg/h returned non ok status:500, message:Server Error
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:53873/eg/h returned non ok status:500, message:Server Error
at __randomizedtesting.SeedInfo.seed([5D1E545638DF3932:DCF8DA4E4F80590E]:0)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:228)
at org.apache.solr.cloud.ShardSplitTest.doTest(ShardSplitTest.java:150)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:680)
Build Log:
[...truncated 9410 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ShardSplitTest
[junit4:junit4] 2> 3612465 T7015 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /eg/h
[junit4:junit4] 2> 3612472 T7015 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solrtest-ShardSplitTest-1371333323952
[junit4:junit4] 2> 3612474 T7015 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 3612476 T7016 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 3612581 T7015 oasc.ZkTestServer.run start zk server on port:53866
[junit4:junit4] 2> 3612585 T7015 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3612596 T7022 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@24f30fc1 name:ZooKeeperConnection Watcher:127.0.0.1:53866 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3612597 T7015 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3612598 T7015 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 3612614 T7015 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3612621 T7024 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2beb4d02 name:ZooKeeperConnection Watcher:127.0.0.1:53866/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3612622 T7015 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3612623 T7015 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 3612633 T7015 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 3612644 T7015 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 3612651 T7015 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 3612663 T7015 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 3612665 T7015 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 3612684 T7015 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 3612686 T7015 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 3612698 T7015 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 3612700 T7015 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 3612710 T7015 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 3612712 T7015 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 3612722 T7015 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 3612725 T7015 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 3612735 T7015 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 3612737 T7015 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 3612748 T7015 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 3612750 T7015 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 3612761 T7015 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 3612763 T7015 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 3612775 T7015 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 3612777 T7015 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 3613396 T7015 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 3613409 T7015 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53869
[junit4:junit4] 2> 3613411 T7015 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 3613412 T7015 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 3613413 T7015 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-controljetty-1371333324271
[junit4:junit4] 2> 3613413 T7015 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-controljetty-1371333324271/solr.xml
[junit4:junit4] 2> 3613414 T7015 oasc.CoreContainer.<init> New CoreContainer 1836434547
[junit4:junit4] 2> 3613415 T7015 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-controljetty-1371333324271/'
[junit4:junit4] 2> 3613416 T7015 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-controljetty-1371333324271/'
[junit4:junit4] 2> 3613611 T7015 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 3613611 T7015 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 3613612 T7015 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 3613612 T7015 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 3613613 T7015 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 3613613 T7015 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 3613614 T7015 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 3613614 T7015 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 3613614 T7015 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 3613615 T7015 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 3613663 T7015 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 3613664 T7015 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53866/solr
[junit4:junit4] 2> 3613665 T7015 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 3613666 T7015 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3613673 T7035 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@200aced1 name:ZooKeeperConnection Watcher:127.0.0.1:53866 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3613674 T7015 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3613679 T7015 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 3613715 T7015 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3613719 T7037 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11529410 name:ZooKeeperConnection Watcher:127.0.0.1:53866/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3613720 T7015 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3613724 T7015 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 3613734 T7015 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 3613743 T7015 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 3613748 T7015 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53869_eg%2Fh
[junit4:junit4] 2> 3613751 T7015 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53869_eg%2Fh
[junit4:junit4] 2> 3613760 T7015 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 3613773 T7015 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 3613780 T7015 oasc.Overseer.start Overseer (id=89871700719108099-127.0.0.1:53869_eg%2Fh-n_0000000000) starting
[junit4:junit4] 2> 3613789 T7015 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 3613801 T7039 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 3613802 T7015 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 3613810 T7015 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 3613815 T7015 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 3613825 T7038 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 3613845 T7040 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 3613846 T7040 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 3615337 T7038 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3615339 T7038 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53869_eg%2Fh",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53869/eg/h"}
[junit4:junit4] 2> 3615339 T7038 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 3615340 T7038 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 3615353 T7037 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 3615852 T7040 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-controljetty-1371333324271/collection1
[junit4:junit4] 2> 3615852 T7040 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 3615854 T7040 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 3615854 T7040 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 3615858 T7040 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-controljetty-1371333324271/collection1/'
[junit4:junit4] 2> 3615859 T7040 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-controljetty-1371333324271/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 3615860 T7040 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-controljetty-1371333324271/collection1/lib/README' to classloader
[junit4:junit4] 2> 3615940 T7040 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 3616053 T7040 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 3616056 T7040 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 3616072 T7040 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 3617162 T7040 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 3617167 T7040 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 3617172 T7040 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 3617184 T7040 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 3617284 T7040 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 3617285 T7040 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-controljetty-1371333324271/collection1/, dataDir=./org.apache.solr.cloud.ShardSplitTest-1371333323951/control/data/
[junit4:junit4] 2> 3617286 T7040 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@60dc3b82
[junit4:junit4] 2> 3617287 T7040 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 3617288 T7040 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371333323951/control/data
[junit4:junit4] 2> 3617289 T7040 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ShardSplitTest-1371333323951/control/data/index/
[junit4:junit4] 2> 3617289 T7040 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ShardSplitTest-1371333323951/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 3617290 T7040 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371333323951/control/data/index
[junit4:junit4] 2> 3617297 T7040 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1568453d lockFactory=org.apache.lucene.store.NativeFSLockFactory@22af989e),segFN=segments_1,generation=1}
[junit4:junit4] 2> 3617298 T7040 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 3617305 T7040 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 3617306 T7040 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 3617308 T7040 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 3617309 T7040 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 3617311 T7040 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 3617311 T7040 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 3617312 T7040 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 3617313 T7040 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 3617314 T7040 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 3617330 T7040 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 3617346 T7040 oass.SolrIndexSearcher.<init> Opening Searcher@db74637 main
[junit4:junit4] 2> 3617347 T7040 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ShardSplitTest-1371333323951/control/data/tlog
[junit4:junit4] 2> 3617349 T7040 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 3617349 T7040 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 3617355 T7041 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@db74637 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 3617357 T7040 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 3617357 T7040 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53869/eg/h collection:control_collection shard:shard1
[junit4:junit4] 2> 3617360 T7040 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 3617387 T7040 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 3617394 T7040 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 3617395 T7040 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 3617395 T7040 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53869/eg/h/collection1/
[junit4:junit4] 2> 3617396 T7040 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 3617396 T7040 oasc.SyncStrategy.syncToMe http://127.0.0.1:53869/eg/h/collection1/ has no replicas
[junit4:junit4] 2> 3617396 T7040 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53869/eg/h/collection1/
[junit4:junit4] 2> 3617397 T7040 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 3618389 T7038 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3618414 T7037 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 3618446 T7040 oasc.ZkController.register We are http://127.0.0.1:53869/eg/h/collection1/ and leader is http://127.0.0.1:53869/eg/h/collection1/
[junit4:junit4] 2> 3618447 T7040 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53869/eg/h
[junit4:junit4] 2> 3618447 T7040 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 3618448 T7040 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 3618448 T7040 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 3618453 T7040 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3618457 T7015 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4] 2> 3618457 T7015 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 3618458 T7015 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 3618513 T7015 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 3618517 T7015 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3618524 T7044 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1646f11a name:ZooKeeperConnection Watcher:127.0.0.1:53866/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3618525 T7015 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3618529 T7015 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 3618538 T7015 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 3619033 T7015 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 3619041 T7015 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53873
[junit4:junit4] 2> 3619045 T7015 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 3619045 T7015 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 3619045 T7015 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty1-1371333330017
[junit4:junit4] 2> 3619046 T7015 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty1-1371333330017/solr.xml
[junit4:junit4] 2> 3619046 T7015 oasc.CoreContainer.<init> New CoreContainer 1968806121
[junit4:junit4] 2> 3619047 T7015 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty1-1371333330017/'
[junit4:junit4] 2> 3619047 T7015 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty1-1371333330017/'
[junit4:junit4] 2> 3619209 T7015 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 3619210 T7015 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 3619211 T7015 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 3619211 T7015 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 3619212 T7015 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 3619212 T7015 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 3619213 T7015 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 3619213 T7015 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 3619214 T7015 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 3619214 T7015 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 3619258 T7015 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 3619259 T7015 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53866/solr
[junit4:junit4] 2> 3619260 T7015 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 3619262 T7015 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3619267 T7055 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@106d975d name:ZooKeeperConnection Watcher:127.0.0.1:53866 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3619269 T7015 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3619273 T7015 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 3619315 T7015 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3619320 T7057 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@61e88827 name:ZooKeeperConnection Watcher:127.0.0.1:53866/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3619321 T7015 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3619332 T7015 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 3619933 T7038 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3619935 T7038 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53869_eg%2Fh",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53869/eg/h"}
[junit4:junit4] 2> 3619946 T7044 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 3619946 T7037 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 3619947 T7057 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 3620342 T7015 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53873_eg%2Fh
[junit4:junit4] 2> 3620344 T7015 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53873_eg%2Fh
[junit4:junit4] 2> 3620349 T7037 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 3620352 T7044 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 3620352 T7044 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 3620352 T7057 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 3620353 T7057 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 3620355 T7037 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 3620369 T7058 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 3620370 T7058 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 3621464 T7038 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3621466 T7038 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53873_eg%2Fh",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53873/eg/h"}
[junit4:junit4] 2> 3621466 T7038 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4] 2> 3621467 T7038 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 3621478 T7044 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 3621478 T7037 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 3621478 T7057 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 3622375 T7058 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty1-1371333330017/collection1
[junit4:junit4] 2> 3622376 T7058 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 3622377 T7058 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 3622378 T7058 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 3622382 T7058 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty1-1371333330017/collection1/'
[junit4:junit4] 2> 3622384 T7058 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty1-1371333330017/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 3622385 T7058 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty1-1371333330017/collection1/lib/README' to classloader
[junit4:junit4] 2> 3622481 T7058 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 3622582 T7058 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 3622586 T7058 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 3622601 T7058 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 3623683 T7058 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 3623690 T7058 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 3623693 T7058 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 3623702 T7058 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 3623774 T7058 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 3623774 T7058 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty1-1371333330017/collection1/, dataDir=./org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty1/
[junit4:junit4] 2> 3623775 T7058 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@60dc3b82
[junit4:junit4] 2> 3623776 T7058 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 3623776 T7058 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty1
[junit4:junit4] 2> 3623777 T7058 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty1/index/
[junit4:junit4] 2> 3623777 T7058 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 3623779 T7058 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty1/index
[junit4:junit4] 2> 3623784 T7058 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@28d8c406 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4299612b),segFN=segments_1,generation=1}
[junit4:junit4] 2> 3623784 T7058 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 3623789 T7058 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 3623790 T7058 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 3623791 T7058 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 3623792 T7058 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 3623793 T7058 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 3623794 T7058 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 3623794 T7058 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 3623795 T7058 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 3623797 T7058 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 3623805 T7058 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 3623813 T7058 oass.SolrIndexSearcher.<init> Opening Searcher@366309eb main
[junit4:junit4] 2> 3623813 T7058 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty1/tlog
[junit4:junit4] 2> 3623815 T7058 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 3623815 T7058 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 3623821 T7059 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@366309eb main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 3623824 T7058 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 3623824 T7058 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53873/eg/h collection:collection1 shard:shard1
[junit4:junit4] 2> 3623827 T7058 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 3623847 T7058 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 3623853 T7058 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 3623854 T7058 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 3623854 T7058 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53873/eg/h/collection1/
[junit4:junit4] 2> 3623854 T7058 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 3623855 T7058 oasc.SyncStrategy.syncToMe http://127.0.0.1:53873/eg/h/collection1/ has no replicas
[junit4:junit4] 2> 3623855 T7058 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53873/eg/h/collection1/
[junit4:junit4] 2> 3623855 T7058 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 3624499 T7038 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3624524 T7044 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 3624525 T7037 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 3624525 T7057 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 3624541 T7058 oasc.ZkController.register We are http://127.0.0.1:53873/eg/h/collection1/ and leader is http://127.0.0.1:53873/eg/h/collection1/
[junit4:junit4] 2> 3624542 T7058 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53873/eg/h
[junit4:junit4] 2> 3624542 T7058 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 3624542 T7058 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 3624542 T7058 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 3624546 T7058 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3624550 T7015 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4] 2> 3624551 T7015 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 3624551 T7015 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 3625074 T7015 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 3625081 T7015 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53876
[junit4:junit4] 2> 3625085 T7015 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 3625086 T7015 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 3625086 T7015 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty2-1371333336086
[junit4:junit4] 2> 3625087 T7015 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty2-1371333336086/solr.xml
[junit4:junit4] 2> 3625088 T7015 oasc.CoreContainer.<init> New CoreContainer 1713490055
[junit4:junit4] 2> 3625089 T7015 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty2-1371333336086/'
[junit4:junit4] 2> 3625089 T7015 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty2-1371333336086/'
[junit4:junit4] 2> 3625272 T7015 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 3625273 T7015 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 3625274 T7015 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 3625274 T7015 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 3625275 T7015 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 3625276 T7015 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 3625276 T7015 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 3625277 T7015 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 3625277 T7015 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 3625278 T7015 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 3625324 T7015 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 3625325 T7015 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53866/solr
[junit4:junit4] 2> 3625326 T7015 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 3625328 T7015 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3625333 T7071 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@77c0f370 name:ZooKeeperConnection Watcher:127.0.0.1:53866 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3625334 T7015 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3625339 T7015 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 3625381 T7015 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3625386 T7073 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@48da6e65 name:ZooKeeperConnection Watcher:127.0.0.1:53866/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3625387 T7015 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3625398 T7015 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 3626045 T7038 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3626047 T7038 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53873_eg%2Fh",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53873/eg/h"}
[junit4:junit4] 2> 3626058 T7037 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 3626058 T7073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 3626058 T7057 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 3626058 T7044 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 3626408 T7015 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53876_eg%2Fh
[junit4:junit4] 2> 3626412 T7015 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53876_eg%2Fh
[junit4:junit4] 2> 3626424 T7037 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 3626427 T7044 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 3626427 T7073 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 3626427 T7044 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 3626427 T7057 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 3626429 T7057 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 3626428 T7073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 3626434 T7037 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 3626449 T7074 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 3626450 T7074 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 3627575 T7038 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3627576 T7038 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53876_eg%2Fh",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53876/eg/h"}
[junit4:junit4] 2> 3627577 T7038 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 3627577 T7038 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 3627588 T7037 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 3627588 T7073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 3627588 T7044 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 3627588 T7057 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 3628459 T7074 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty2-1371333336086/collection1
[junit4:junit4] 2> 3628460 T7074 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 3628461 T7074 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 3628462 T7074 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 3628465 T7074 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty2-1371333336086/collection1/'
[junit4:junit4] 2> 3628467 T7074 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty2-1371333336086/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 3628467 T7074 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty2-1371333336086/collection1/lib/README' to classloader
[junit4:junit4] 2> 3628554 T7074 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 3628663 T7074 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 3628667 T7074 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 3628684 T7074 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 3629711 T7074 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 3629716 T7074 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 3629719 T7074 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 3629729 T7074 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 3629799 T7074 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 3629800 T7074 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty2-1371333336086/collection1/, dataDir=./org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty2/
[junit4:junit4] 2> 3629801 T7074 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@60dc3b82
[junit4:junit4] 2> 3629802 T7074 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 3629804 T7074 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty2
[junit4:junit4] 2> 3629805 T7074 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty2/index/
[junit4:junit4] 2> 3629806 T7074 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 3629807 T7074 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty2/index
[junit4:junit4] 2> 3629813 T7074 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@54f6cbf8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@64b889c5),segFN=segments_1,generation=1}
[junit4:junit4] 2> 3629813 T7074 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 3629818 T7074 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 3629818 T7074 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 3629819 T7074 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 3629820 T7074 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 3629822 T7074 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 3629822 T7074 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 3629823 T7074 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 3629824 T7074 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 3629825 T7074 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 3629837 T7074 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 3629847 T7074 oass.SolrIndexSearcher.<init> Opening Searcher@16f4f48a main
[junit4:junit4] 2> 3629848 T7074 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty2/tlog
[junit4:junit4] 2> 3629849 T7074 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 3629850 T7074 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 3629876 T7075 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@16f4f48a main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 3629877 T7074 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 3629878 T7074 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53876/eg/h collection:collection1 shard:shard2
[junit4:junit4] 2> 3629880 T7074 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 3629900 T7074 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 3629908 T7074 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 3629908 T7074 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 3629909 T7074 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53876/eg/h/collection1/
[junit4:junit4] 2> 3629909 T7074 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 3629910 T7074 oasc.SyncStrategy.syncToMe http://127.0.0.1:53876/eg/h/collection1/ has no replicas
[junit4:junit4] 2> 3629910 T7074 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53876/eg/h/collection1/
[junit4:junit4] 2> 3629912 T7074 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 3630612 T7038 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3630635 T7037 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 3630636 T7073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 3630637 T7044 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 3630636 T7057 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 3630645 T7074 oasc.ZkController.register We are http://127.0.0.1:53876/eg/h/collection1/ and leader is http://127.0.0.1:53876/eg/h/collection1/
[junit4:junit4] 2> 3630645 T7074 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53876/eg/h
[junit4:junit4] 2> 3630646 T7074 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 3630646 T7074 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 3630647 T7074 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 3630651 T7074 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3630654 T7015 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4] 2> 3630654 T7015 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 3630655 T7015 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 3631174 T7015 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 3631181 T7015 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53879
[junit4:junit4] 2> 3631186 T7015 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 3631187 T7015 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 3631188 T7015 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1371333342179
[junit4:junit4] 2> 3631188 T7015 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1371333342179/solr.xml
[junit4:junit4] 2> 3631189 T7015 oasc.CoreContainer.<init> New CoreContainer 2113529338
[junit4:junit4] 2> 3631190 T7015 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1371333342179/'
[junit4:junit4] 2> 3631190 T7015 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1371333342179/'
[junit4:junit4] 2> 3631361 T7015 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 3631361 T7015 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 3631362 T7015 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 3631362 T7015 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 3631363 T7015 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 3631363 T7015 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 3631364 T7015 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 3631364 T7015 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 3631365 T7015 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 3631365 T7015 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 3631407 T7015 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 3631407 T7015 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53866/solr
[junit4:junit4] 2> 3631408 T7015 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 3631410 T7015 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3631418 T7087 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5950de91 name:ZooKeeperConnection Watcher:127.0.0.1:53866 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3631420 T7015 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3631426 T7015 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 3631476 T7015 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3631481 T7089 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6f55e4fd name:ZooKeeperConnection Watcher:127.0.0.1:53866/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3631482 T7015 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3631494 T7015 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 3632157 T7038 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3632159 T7038 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53876_eg%2Fh",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53876/eg/h"}
[junit4:junit4] 2> 3632170 T7037 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 3632170 T7073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 3632171 T7044 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 3632171 T7089 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 3632171 T7057 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 3632505 T7015 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53879_eg%2Fh
[junit4:junit4] 2> 3632508 T7015 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53879_eg%2Fh
[junit4:junit4] 2> 3632515 T7037 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 3632518 T7089 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 3632519 T7089 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 3632520 T7073 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 3632520 T7073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 3632520 T7057 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 3632520 T7057 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 3632520 T7044 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 3632522 T7044 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 3632526 T7037 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 3632540 T7090 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 3632541 T7090 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 3633688 T7038 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3633690 T7038 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53879_eg%2Fh",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53879/eg/h"}
[junit4:junit4] 2> 3633691 T7038 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 3633691 T7038 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 3633706 T7037 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 3633707 T7044 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 3633707 T7057 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 3633706 T7089 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 3633706 T7073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 3634547 T7090 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1371333342179/collection1
[junit4:junit4] 2> 3634548 T7090 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 3634550 T7090 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 3634551 T7090 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 3634556 T7090 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1371333342179/collection1/'
[junit4:junit4] 2> 3634560 T7090 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty3-1371333342179/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 3634562 T7090 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty3-1371333342179/collection1/lib/README' to classloader
[junit4:junit4] 2> 3634673 T7090 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 3634781 T7090 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 3634786 T7090 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 3634816 T7090 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 3635863 T7090 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 3635868 T7090 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 3635872 T7090 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 3635881 T7090 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 3635981 T7090 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 3635982 T7090 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1371333342179/collection1/, dataDir=./org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty3/
[junit4:junit4] 2> 3635982 T7090 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@60dc3b82
[junit4:junit4] 2> 3635984 T7090 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 3635985 T7090 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty3
[junit4:junit4] 2> 3635986 T7090 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty3/index/
[junit4:junit4] 2> 3635986 T7090 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 3635989 T7090 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty3/index
[junit4:junit4] 2> 3635995 T7090 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@15f32451 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5dfd0f27),segFN=segments_1,generation=1}
[junit4:junit4] 2> 3635996 T7090 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 3636001 T7090 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 3636002 T7090 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 3636003 T7090 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 3636005 T7090 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 3636007 T7090 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 3636007 T7090 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 3636008 T7090 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 3636010 T7090 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 3636011 T7090 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 3636024 T7090 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 3636036 T7090 oass.SolrIndexSearcher.<init> Opening Searcher@71453c38 main
[junit4:junit4] 2> 3636037 T7090 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty3/tlog
[junit4:junit4] 2> 3636039 T7090 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 3636039 T7090 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 3636050 T7091 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@71453c38 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 3636055 T7090 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 3636056 T7090 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53879/eg/h collection:collection1 shard:shard1
[junit4:junit4] 2> 3636065 T7090 oasc.ZkController.register We are http://127.0.0.1:53879/eg/h/collection1/ and leader is http://127.0.0.1:53873/eg/h/collection1/
[junit4:junit4] 2> 3636066 T7090 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53879/eg/h
[junit4:junit4] 2> 3636066 T7090 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 3636066 T7090 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C9449 name=collection1 org.apache.solr.core.SolrCore@415cfff url=http://127.0.0.1:53879/eg/h/collection1 node=127.0.0.1:53879_eg%2Fh C9449_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:53879_eg%2Fh, base_url=http://127.0.0.1:53879/eg/h}
[junit4:junit4] 2> 3636070 T7092 C9449 P53879 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 3636073 T7092 C9449 P53879 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 3636073 T7092 C9449 P53879 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 3636073 T7092 C9449 P53879 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 3636076 T7090 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3636078 T7092 C9449 P53879 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 3636108 T7015 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4] 2> 3636109 T7015 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 3636110 T7015 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 3636182 T7053 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 3636730 T7038 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3636732 T7038 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53879_eg%2Fh",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53879/eg/h"}
[junit4:junit4] 2> 3636742 T7037 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 3636742 T7057 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 3636743 T7089 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 3636743 T7044 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 3636743 T7073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 3636792 T7015 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 3636818 T7015 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53883
[junit4:junit4] 2> 3636820 T7015 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 3636821 T7015 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 3636823 T7015 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty4-1371333347664
[junit4:junit4] 2> 3636825 T7015 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty4-1371333347664/solr.xml
[junit4:junit4] 2> 3636827 T7015 oasc.CoreContainer.<init> New CoreContainer 4166711
[junit4:junit4] 2> 3636830 T7015 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty4-1371333347664/'
[junit4:junit4] 2> 3636831 T7015 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty4-1371333347664/'
[junit4:junit4] 2> 3637076 T7015 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 3637077 T7015 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 3637077 T7015 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 3637078 T7015 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 3637079 T7015 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 3637079 T7015 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 3637080 T7015 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 3637080 T7015 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 3637081 T7015 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 3637082 T7015 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 3637128 T7015 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 3637130 T7015 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53866/solr
[junit4:junit4] 2> 3637130 T7015 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 3637132 T7015 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3637139 T7104 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@de5745d name:ZooKeeperConnection Watcher:127.0.0.1:53866 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3637141 T7015 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3637146 T7015 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 3637184 T7053 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 3637187 T7053 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=4&state=recovering&nodeName=127.0.0.1:53879_eg%252Fh&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1006
[junit4:junit4] 2> 3637203 T7015 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3637213 T7106 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1766949c name:ZooKeeperConnection Watcher:127.0.0.1:53866/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3637214 T7015 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3637243 T7015 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 3638257 T7015 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53883_eg%2Fh
[junit4:junit4] 2> 3638262 T7015 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53883_eg%2Fh
[junit4:junit4] 2> 3638271 T7037 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 3638277 T7106 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 3638278 T7073 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 3638278 T7073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 3638279 T7057 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 3638279 T7057 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 3638280 T7044 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 3638281 T7044 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 3638281 T7089 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 3638282 T7089 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 3638289 T7037 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 3638307 T7107 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 3638307 T7107 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> ASYNC NEW_CORE C9450 name=collection1 org.apache.solr.core.SolrCore@415cfff url=http://127.0.0.1:53879/eg/h/collection1 node=127.0.0.1:53879_eg%2Fh C9450_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:53879_eg%2Fh, base_url=http://127.0.0.1:53879/eg/h}
[junit4:junit4] 2> 3639191 T7092 C9450 P53879 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:53873/eg/h/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 3639191 T7092 C9450 P53879 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:53879/eg/h START replicas=[http://127.0.0.1:53873/eg/h/collection1/] nUpdates=100
[junit4:junit4] 2> 3639192 T7092 C9450 P53879 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 3639202 T7092 C9450 P53879 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 3639202 T7092 C9450 P53879 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 3639203 T7092 C9450 P53879 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 3639205 T7092 C9450 P53879 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 3639207 T7092 C9450 P53879 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:53873/eg/h/collection1/. core=collection1
[junit4:junit4] 2> 3639207 T7092 C9450 P53879 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C9451 name=collection1 org.apache.solr.core.SolrCore@40aed2d url=http://127.0.0.1:53873/eg/h/collection1 node=127.0.0.1:53873_eg%2Fh C9451_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:53873_eg%2Fh, base_url=http://127.0.0.1:53873/eg/h, leader=true}
[junit4:junit4] 2> 3639222 T7050 C9451 P53873 oasc.SolrCore.execute [collection1] webapp=/eg/h path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=2
[junit4:junit4] 2> 3639273 T7051 C9451 P53873 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 3639276 T7051 C9451 P53873 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@28d8c406 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4299612b),segFN=segments_1,generation=1}
[junit4:junit4] 2> 3639278 T7051 C9451 P53873 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 3639280 T7051 C9451 P53873 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@28d8c406 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4299612b),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@28d8c406 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4299612b),segFN=segments_2,generation=2}
[junit4:junit4] 2> 3639281 T7051 C9451 P53873 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 3639282 T7051 C9451 P53873 oass.SolrIndexSearcher.<init> Opening Searcher@2c96066d realtime
[junit4:junit4] 2> 3639283 T7051 C9451 P53873 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 3639284 T7051 C9451 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 11
[junit4:junit4] 2> 3639286 T7092 C9450 P53879 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 3639287 T7092 C9450 P53879 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 3639294 T7048 C9451 P53873 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 3639294 T7048 C9451 P53873 oasc.SolrCore.execute [collection1] webapp=/eg/h path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2
[junit4:junit4] 2> 3639296 T7092 C9450 P53879 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 3639296 T7092 C9450 P53879 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 3639296 T7092 C9450 P53879 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 3639299 T7049 C9451 P53873 oasc.SolrCore.execute [collection1] webapp=/eg/h path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 3639300 T7092 C9450 P53879 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 3639302 T7092 C9450 P53879 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty3/index.20130615155550779
[junit4:junit4] 2> 3639302 T7092 C9450 P53879 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@65fd9e3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@33ba3e7a) fullCopy=false
[junit4:junit4] 2> 3639306 T7053 C9451 P53873 oasc.SolrCore.execute [collection1] webapp=/eg/h path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1
[junit4:junit4] 2> 3639308 T7092 C9450 P53879 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 3639310 T7092 C9450 P53879 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 3639310 T7092 C9450 P53879 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 3639312 T7092 C9450 P53879 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@15f32451 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5dfd0f27),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@15f32451 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5dfd0f27),segFN=segments_2,generation=2}
[junit4:junit4] 2> 3639313 T7092 C9450 P53879 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 3639313 T7092 C9450 P53879 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 3639313 T7092 C9450 P53879 oass.SolrIndexSearcher.<init> Opening Searcher@438c9745 main
[junit4:junit4] 2> 3639315 T7091 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@438c9745 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 3639316 T7092 C9450 P53879 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty3/index.20130615155550779 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty3/index.20130615155550779;done=true>>]
[junit4:junit4] 2> 3639316 T7092 C9450 P53879 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty3/index.20130615155550779
[junit4:junit4] 2> 3639317 T7092 C9450 P53879 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty3/index.20130615155550779
[junit4:junit4] 2> 3639317 T7092 C9450 P53879 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 3639318 T7092 C9450 P53879 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 3639318 T7092 C9450 P53879 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 3639318 T7092 C9450 P53879 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 3639321 T7092 C9450 P53879 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 3639776 T7038 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3639778 T7038 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53883_eg%2Fh",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53883/eg/h"}
[junit4:junit4] 2> 3639779 T7038 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 3639780 T7038 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 3639791 T7038 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53879_eg%2Fh",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53879/eg/h"}
[junit4:junit4] 2> 3639804 T7073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 3639805 T7044 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 3639805 T7057 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 3639805 T7037 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 3639805 T7089 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 3639805 T7106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 3640323 T7107 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty4-1371333347664/collection1
[junit4:junit4] 2> 3640323 T7107 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 3640325 T7107 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 3640326 T7107 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 3640331 T7107 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty4-1371333347664/collection1/'
[junit4:junit4] 2> 3640332 T7107 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty4-1371333347664/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 3640333 T7107 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty4-1371333347664/collection1/lib/README' to classloader
[junit4:junit4] 2> 3640425 T7107 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 3640534 T7107 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 3640538 T7107 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 3640547 T7107 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 3641489 T7107 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 3641494 T7107 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 3641499 T7107 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 3641508 T7107 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 3641579 T7107 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 3641580 T7107 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty4-1371333347664/collection1/, dataDir=./org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty4/
[junit4:junit4] 2> 3641580 T7107 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@60dc3b82
[junit4:junit4] 2> 3641581 T7107 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 3641582 T7107 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty4
[junit4:junit4] 2> 3641582 T7107 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty4/index/
[junit4:junit4] 2> 3641582 T7107 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 3641584 T7107 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty4/index
[junit4:junit4] 2> 3641590 T7107 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@41a146c lockFactory=org.apache.lucene.store.NativeFSLockFactory@1c2151b0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 3641591 T7107 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 3641596 T7107 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 3641596 T7107 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 3641597 T7107 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 3641599 T7107 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 3641600 T7107 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 3641600 T7107 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 3641602 T7107 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 3641603 T7107 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 3641603 T7107 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 3641615 T7107 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 3641625 T7107 oass.SolrIndexSearcher.<init> Opening Searcher@7ea10214 main
[junit4:junit4] 2> 3641625 T7107 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty4/tlog
[junit4:junit4] 2> 3641627 T7107 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 3641627 T7107 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 3641635 T7110 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7ea10214 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 3641639 T7107 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 3641639 T7107 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53883/eg/h collection:collection1 shard:shard2
[junit4:junit4] 2> 3641649 T7107 oasc.ZkController.register We are http://127.0.0.1:53883/eg/h/collection1/ and leader is http://127.0.0.1:53876/eg/h/collection1/
[junit4:junit4] 2> 3641650 T7107 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53883/eg/h
[junit4:junit4] 2> 3641650 T7107 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 3641650 T7107 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C9452 name=collection1 org.apache.solr.core.SolrCore@392439c3 url=http://127.0.0.1:53883/eg/h/collection1 node=127.0.0.1:53883_eg%2Fh C9452_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:53883_eg%2Fh, base_url=http://127.0.0.1:53883/eg/h}
[junit4:junit4] 2> 3641652 T7111 C9452 P53883 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 3641653 T7111 C9452 P53883 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 3641653 T7107 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3641653 T7111 C9452 P53883 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 3641654 T7111 C9452 P53883 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 3641657 T7015 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4] 2> 3641657 T7111 C9452 P53883 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 3641658 T7015 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 3641659 T7015 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 3641714 T7015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3641717 T7067 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 3641718 T7015 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 3641720 T7015 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 3641722 T7015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3642729 T7015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3642839 T7038 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3642841 T7038 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53883_eg%2Fh",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53883/eg/h"}
[junit4:junit4] 2> 3642855 T7073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 3642856 T7044 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 3642855 T7037 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 3642856 T7057 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 3642855 T7106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 3642856 T7089 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 3643720 T7067 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 3643721 T7067 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=5&state=recovering&nodeName=127.0.0.1:53883_eg%252Fh&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2004
[junit4:junit4] 2> 3643737 T7015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3644742 T7015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> C9452_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:53883_eg%2Fh, base_url=http://127.0.0.1:53883/eg/h}
[junit4:junit4] 2> 3645726 T7111 C9452 P53883 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:53876/eg/h/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 3645726 T7111 C9452 P53883 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:53883/eg/h START replicas=[http://127.0.0.1:53876/eg/h/collection1/] nUpdates=100
[junit4:junit4] 2> 3645728 T7111 C9452 P53883 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 3645728 T7111 C9452 P53883 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 3645729 T7111 C9452 P53883 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 3645731 T7111 C9452 P53883 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 3645731 T7111 C9452 P53883 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 3645733 T7111 C9452 P53883 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:53876/eg/h/collection1/. core=collection1
[junit4:junit4] 2> 3645734 T7111 C9452 P53883 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C9453 name=collection1 org.apache.solr.core.SolrCore@13563135 url=http://127.0.0.1:53876/eg/h/collection1 node=127.0.0.1:53876_eg%2Fh C9453_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:53876_eg%2Fh, base_url=http://127.0.0.1:53876/eg/h, leader=true}
[junit4:junit4] 2> 3645746 T7068 C9453 P53876 oasc.SolrCore.execute [collection1] webapp=/eg/h path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 3645749 T7015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3645798 T7063 C9453 P53876 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 3645802 T7063 C9453 P53876 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@54f6cbf8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@64b889c5),segFN=segments_1,generation=1}
[junit4:junit4] 2> 3645804 T7063 C9453 P53876 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 3645810 T7063 C9453 P53876 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@54f6cbf8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@64b889c5),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@54f6cbf8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@64b889c5),segFN=segments_2,generation=2}
[junit4:junit4] 2> 3645811 T7063 C9453 P53876 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 3645813 T7063 C9453 P53876 oass.SolrIndexSearcher.<init> Opening Searcher@513a664f realtime
[junit4:junit4] 2> 3645814 T7063 C9453 P53876 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 3645815 T7063 C9453 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 18
[junit4:junit4] 2> 3645818 T7111 C9452 P53883 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 3645819 T7111 C9452 P53883 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 3645827 T7065 C9453 P53876 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 3645828 T7065 C9453 P53876 oasc.SolrCore.execute [collection1] webapp=/eg/h path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2
[junit4:junit4] 2> 3645830 T7111 C9452 P53883 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 3645831 T7111 C9452 P53883 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 3645832 T7111 C9452 P53883 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 3645836 T7066 C9453 P53876 oasc.SolrCore.execute [collection1] webapp=/eg/h path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 3645838 T7111 C9452 P53883 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 3645841 T7111 C9452 P53883 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty4/index.20130615155557317
[junit4:junit4] 2> 3645841 T7111 C9452 P53883 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@4c6253ac lockFactory=org.apache.lucene.store.NativeFSLockFactory@593bd746) fullCopy=false
[junit4:junit4] 2> 3645846 T7067 C9453 P53876 oasc.SolrCore.execute [collection1] webapp=/eg/h path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 3645849 T7111 C9452 P53883 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 3645851 T7111 C9452 P53883 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 3645852 T7111 C9452 P53883 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 3645856 T7111 C9452 P53883 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@41a146c lockFactory=org.apache.lucene.store.NativeFSLockFactory@1c2151b0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@41a146c lockFactory=org.apache.lucene.store.NativeFSLockFactory@1c2151b0),segFN=segments_2,generation=2}
[junit4:junit4] 2> 3645858 T7111 C9452 P53883 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 3645859 T7111 C9452 P53883 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 3645859 T7111 C9452 P53883 oass.SolrIndexSearcher.<init> Opening Searcher@67acc2d4 main
[junit4:junit4] 2> 3645862 T7110 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@67acc2d4 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 3645863 T7111 C9452 P53883 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty4/index.20130615155557317 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty4/index.20130615155557317;done=true>>]
[junit4:junit4] 2> 3645864 T7111 C9452 P53883 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty4/index.20130615155557317
[junit4:junit4] 2> 3645864 T7111 C9452 P53883 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.ShardSplitTest-1371333323951/jetty4/index.20130615155557317
[junit4:junit4] 2> 3645865 T7111 C9452 P53883 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 3645865 T7111 C9452 P53883 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 3645866 T7111 C9452 P53883 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 3645866 T7111 C9452 P53883 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 3645870 T7111 C9452 P53883 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 3645888 T7038 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3645891 T7038 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53883_eg%2Fh",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53883/eg/h"}
[junit4:junit4] 2> 3645908 T7037 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 3645908 T7106 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 3645909 T7089 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 3645908 T7044 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 3645909 T7073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 3645908 T7057 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 3646755 T7015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3646757 T7015 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> ASYNC NEW_CORE C9454 name=collection1 org.apache.solr.core.SolrCore@5706ca5 url=http://127.0.0.1:53869/eg/h/collection1 node=127.0.0.1:53869_eg%2Fh C9454_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:53869_eg%2Fh, base_url=http://127.0.0.1:53869/eg/h, leader=true}
[junit4:junit4] 2> 3646784 T7030 C9454 P53869 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 3646787 T7030 C9454 P53869 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1568453d lockFactory=org.apache.lucene.store.NativeFSLockFactory@22af989e),segFN=segments_1,generation=1}
[junit4:junit4] 2> 3646787 T7030 C9454 P53869 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 3646788 T7030 C9454 P53869 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1568453d lockFactory=org.apache.lucene.store.NativeFSLockFactory@22af989e),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1568453d lockFactory=org.apache.lucene.store.NativeFSLockFactory@22af989e),segFN=segments_2,generation=2}
[junit4:junit4] 2> 3646789 T7030 C9454 P53869 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 3646789 T7030 C9454 P53869 oass.SolrIndexSearcher.<init> Opening Searcher@3d57e7d3 main
[junit4:junit4] 2> 3646790 T7030 C9454 P53869 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 3646792 T7041 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3d57e7d3 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 3646793 T7030 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 9
[junit4:junit4] 2> ASYNC NEW_CORE C9455 name=collection1 org.apache.solr.core.SolrCore@40aed2d url=http://127.0.0.1:53873/eg/h/collection1 node=127.0.0.1:53873_eg%2Fh C9455_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:53873_eg%2Fh, base_url=http://127.0.0.1:53873/eg/h, leader=true}
[junit4:junit4] 2> 3646837 T7053 C9455 P53873 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 3646841 T7053 C9455 P53873 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@28d8c406 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4299612b),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@28d8c406 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4299612b),segFN=segments_3,generation=3}
[junit4:junit4] 2> 3646843 T7053 C9455 P53873 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 3646844 T7053 C9455 P53873 oass.SolrIndexSearcher.<init> Opening Searcher@47f6844 main
[junit4:junit4] 2> 3646845 T7053 C9455 P53873 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 3646846 T7059 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@47f6844 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 3646847 T7053 C9455 P53873 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:53879/eg/h/collection1/, StdNode: http://127.0.0.1:53876/eg/h/collection1/, StdNode: http://127.0.0.1:53883/eg/h/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> ASYNC NEW_CORE C9456 name=collection1 org.apache.solr.core.SolrCore@13563135 url=http://127.0.0.1:53876/eg/h/collection1 node=127.0.0.1:53876_eg%2Fh C9456_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:53876_eg%2Fh, base_url=http://127.0.0.1:53876/eg/h, leader=true}
[junit4:junit4] 2> 3646869 T7064 C9456 P53876 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> ASYNC NEW_CORE C9457 name=collection1 org.apache.solr.core.SolrCore@392439c3 url=http://127.0.0.1:53883/eg/h/collection1 node=127.0.0.1:53883_eg%2Fh C9457_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:53883_eg%2Fh, base_url=http://127.0.0.1:53883/eg/h}
[junit4:junit4] 2> 3646901 T7098 C9457 P53883 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> ASYNC NEW_CORE C9458 name=collection1 org.apache.solr.core.SolrCore@415cfff url=http://127.0.0.1:53879/eg/h/collection1 node=127.0.0.1:53879_eg%2Fh C9458_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:53879_eg%2Fh, base_url=http://127.0.0.1:53879/eg/h}
[junit4:junit4] 2> 3646921 T7083 C9458 P53879 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 3646926 T7064 C9456 P53876 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@54f6cbf8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@64b889c5),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@54f6cbf8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@64b889c5),segFN=segments_3,generation=3}
[junit4:junit4] 2> 3646932 T7064 C9456 P53876 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 3646934 T7064 C9456 P53876 oass.SolrIndexSearcher.<init> Opening Searcher@6c951426 main
[junit4:junit4] 2> 3646934 T7064 C9456 P53876 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 3646937 T7075 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6c951426 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 3646937 T7098 C9457 P53883 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@41a146c lockFactory=org.apache.lucene.store.NativeFSLockFactory@1c2151b0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@41a146c lockFactory=org.apache.lucene.store.NativeFSLockFactory@1c2151b0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 3646940 T7098 C9457 P53883 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 3646939 T7064 C9456 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 71
[junit4:junit4] 2> 3646942 T7098 C9457 P53883 oass.SolrIndexSearcher.<init> Opening Searcher@623ac4bd main
[junit4:junit4] 2> 3646944 T7098 C9457 P53883 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 3646940 T7083 C9458 P53879 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@15f32451 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5dfd0f27),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@15f32451 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5dfd0f27),segFN=segments_3,generation=3}
[junit4:junit4] 2> 3646945 T7110 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@623ac4bd main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 3646947 T7083 C9458 P53879 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 3646947 T7098 C9457 P53883 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 46
[junit4:junit4] 2> 3646948 T7083 C9458 P53879 oass.SolrIndexSearcher.<init> Opening Searcher@d53c86e main
[junit4:junit4] 2> 3646949 T7083 C9458 P53879 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 3646952 T7091 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@d53c86e main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 3646953 T7083 C9458 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 32
[junit4:junit4] 2> 3646955 T7053 C9455 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 118
[junit4:junit4] 2> 3646958 T7015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3646968 T7051 C9455 P53873 oasc.SolrCore.execute [collection1] webapp=/eg/h path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 2> 3646977 T7084 C9458 P53879 oasc.SolrCore.execute [collection1] webapp=/eg/h path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 2> 3646985 T7065 C9456 P53876 oasc.SolrCore.execute [collection1] webapp=/eg/h path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 3646993 T7099 C9457 P53883 oasc.SolrCore.execute [collection1] webapp=/eg/h path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 2> 3649004 T7030 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1437947249797627904)} 0 3
[junit4:junit4] 2> 3649022 T7084 C9458 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={update.distrib=FROMLEADER&_version_=-1437947249809162240&update.from=http://127.0.0.1:53873/eg/h/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1437947249809162240)} 0 2
[junit4:junit4] 2> 3649043 T7099 C9457 P53883 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={update.distrib=FROMLEADER&_version_=-1437947249816502272&update.from=http://127.0.0.1:53876/eg/h/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1437947249816502272)} 0 3
[junit4:junit4] 2> 3649045 T7065 C9456 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1437947249816502272)} 0 26
[junit4:junit4] 2> 3649047 T7050 C9455 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1437947249809162240)} 0 35
[junit4:junit4] 2> 3649055 T7031 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[a!0 (1437947249851105280)]} 0 4
[junit4:junit4] 2> 3649074 T7100 C9457 P53883 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[a!0 (1437947249862639616)]} 0 2
[junit4:junit4] 2> 3649076 T7066 C9456 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[a!0 (1437947249862639616)]} 0 13
[junit4:junit4] 2> 3649083 T7032 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[b!1 (1437947249882562560)]} 0 1
[junit4:junit4] 2> 3649107 T7085 C9458 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[b!1 (1437947249902485504)]} 0 2
[junit4:junit4] 2> 3649109 T7051 C9455 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[b!1 (1437947249902485504)]} 0 9
[junit4:junit4] 2> 3649111 T7067 C9456 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[b!1]} 0 19
[junit4:junit4] 2> 3649117 T7033 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[c!2 (1437947249918214144)]} 0 1
[junit4:junit4] 2> 3649126 T7080 C9458 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[c!2 (1437947249923457024)]} 0 1
[junit4:junit4] 2> 3649127 T7052 C9455 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[c!2 (1437947249923457024)]} 0 6
[junit4:junit4] 2> 3649133 T7029 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[d!3 (1437947249934991360)]} 0 1
[junit4:junit4] 2> 3649153 T7101 C9457 P53883 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[d!3 (1437947249951768576)]} 0 1
[junit4:junit4] 2> 3649155 T7068 C9456 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[d!3 (1437947249951768576)]} 0 8
[junit4:junit4] 2> 3649157 T7048 C9455 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[d!3]} 0 15
[junit4:junit4] 2> 3649164 T7028 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[e!4 (1437947249967497216)]} 0 1
[junit4:junit4] 2> 3649181 T7102 C9457 P53883 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[e!4 (1437947249980080128)]} 0 1
[junit4:junit4] 2> 3649183 T7063 C9456 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[e!4 (1437947249980080128)]} 0 8
[junit4:junit4] 2> 3649185 T7049 C9455 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[e!4]} 0 13
[junit4:junit4] 2> 3649193 T7030 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[f!5 (1437947249997905920)]} 0 1
[junit4:junit4] 2> 3649207 T7097 C9457 P53883 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[f!5 (1437947250008391680)]} 0 1
[junit4:junit4] 2> 3649209 T7064 C9456 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[f!5 (1437947250008391680)]} 0 7
[junit4:junit4] 2> 3649210 T7053 C9455 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[f!5]} 0 12
[junit4:junit4] 2> 3649218 T7031 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[g!6 (1437947250025168896)]} 0 1
[junit4:junit4] 2> 3649232 T7081 C9458 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[g!6 (1437947250034606080)]} 0 1
[junit4:junit4] 2> 3649234 T7050 C9455 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[g!6 (1437947250034606080)]} 0 7
[junit4:junit4] 2> 3649237 T7065 C9456 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[g!6]} 0 14
[junit4:junit4] 2> 3649243 T7033 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[h!7 (1437947250050334720)]} 0 1
[junit4:junit4] 2> 3649260 T7082 C9458 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[h!7 (1437947250063966208)]} 0 1
[junit4:junit4] 2> 3649262 T7051 C9455 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[h!7 (1437947250063966208)]} 0 8
[junit4:junit4] 2> 3649264 T7066 C9456 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[h!7]} 0 14
[junit4:junit4] 2> 3649270 T7029 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[i!8 (1437947250078646272)]} 0 1
[junit4:junit4] 2> 3649282 T7083 C9458 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[i!8 (1437947250085986304)]} 0 1
[junit4:junit4] 2> 3649284 T7052 C9455 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[i!8 (1437947250085986304)]} 0 9
[junit4:junit4] 2> 3649290 T7028 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[j!9 (1437947250099617792)]} 0 1
[junit4:junit4] 2> 3649308 T7084 C9458 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[j!9 (1437947250111152128)]} 0 1
[junit4:junit4] 2> 3649310 T7048 C9455 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[j!9 (1437947250111152128)]} 0 11
[junit4:junit4] 2> 3649312 T7067 C9456 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[j!9]} 0 17
[junit4:junit4] 2> 3649323 T7030 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[k!10 (1437947250134220800)]} 0 1
[junit4:junit4] 2> 3649337 T7085 C9458 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[k!10 (1437947250144706560)]} 0 1
[junit4:junit4] 2> 3649339 T7049 C9455 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[k!10 (1437947250144706560)]} 0 6
[junit4:junit4] 2> 3649340 T7068 C9456 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[k!10]} 0 12
[junit4:junit4] 2> 3649348 T7032 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[l!11 (1437947250161483776)]} 0 1
[junit4:junit4] 2> 3649359 T7098 C9457 P53883 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[l!11 (1437947250167775232)]} 0 1
[junit4:junit4] 2> 3649361 T7063 C9456 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[l!11 (1437947250167775232)]} 0 7
[junit4:junit4] 2> 3649368 T7031 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[m!12 (1437947250181406720)]} 0 1
[junit4:junit4] 2> 3649378 T7099 C9457 P53883 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[m!12 (1437947250187698176)]} 0 1
[junit4:junit4] 2> 3649380 T7064 C9456 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[m!12 (1437947250187698176)]} 0 7
[junit4:junit4] 2> 3649386 T7033 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[n!13 (1437947250200281088)]} 0 1
[junit4:junit4] 2> 3649396 T7080 C9458 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[n!13 (1437947250206572544)]} 0 1
[junit4:junit4] 2> 3649398 T7053 C9455 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[n!13 (1437947250206572544)]} 0 7
[junit4:junit4] 2> 3649405 T7029 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[o!14 (1437947250221252608)]} 0 1
[junit4:junit4] 2> 3649426 T7100 C9457 P53883 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[o!14 (1437947250238029824)]} 0 1
[junit4:junit4] 2> 3649427 T7065 C9456 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[o!14 (1437947250238029824)]} 0 6
[junit4:junit4] 2> 3649429 T7051 C9455 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[o!14]} 0 19
[junit4:junit4] 2> 3649434 T7028 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[p!15 (1437947250250612736)]} 0 2
[junit4:junit4] 2> 3649444 T7082 C9458 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[p!15 (1437947250256904192)]} 0 2
[junit4:junit4] 2> 3649445 T7052 C9455 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[p!15 (1437947250256904192)]} 0 7
[junit4:junit4] 2> 3649451 T7030 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[q!16 (1437947250269487104)]} 0 0
[junit4:junit4] 2> 3649461 T7083 C9458 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[q!16 (1437947250274729984)]} 0 0
[junit4:junit4] 2> 3649463 T7048 C9455 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[q!16 (1437947250274729984)]} 0 7
[junit4:junit4] 2> 3649472 T7032 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[r!17 (1437947250291507200)]} 0 0
[junit4:junit4] 2> 3649490 T7102 C9457 P53883 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[r!17 (1437947250306187264)]} 0 1
[junit4:junit4] 2> 3649492 T7067 C9456 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[r!17 (1437947250306187264)]} 0 7
[junit4:junit4] 2> 3649494 T7049 C9455 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[r!17]} 0 12
[junit4:junit4] 2> 3649501 T7031 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[s!18 (1437947250320867328)]} 0 1
[junit4:junit4] 2> 3649512 T7084 C9458 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[s!18 (1437947250328207360)]} 0 1
[junit4:junit4] 2> 3649513 T7050 C9455 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[s!18 (1437947250328207360)]} 0 7
[junit4:junit4] 2> 3649518 T7033 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[t!19 (1437947250339741696)]} 0 1
[junit4:junit4] 2> 3649533 T7085 C9458 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[t!19 (1437947250348130304)]} 0 1
[junit4:junit4] 2> 3649536 T7053 C9455 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[t!19 (1437947250348130304)]} 0 10
[junit4:junit4] 2> 3649546 T7029 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[u!20 (1437947250368053248)]} 0 1
[junit4:junit4] 2> 3649554 T7097 C9457 P53883 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[u!20 (1437947250373296128)]} 0 1
[junit4:junit4] 2> 3649555 T7068 C9456 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[u!20 (1437947250373296128)]} 0 5
[junit4:junit4] 2> 3649560 T7028 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[v!21 (1437947250383781888)]} 0 1
[junit4:junit4] 2> 3649569 T7081 C9458 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[v!21 (1437947250389024768)]} 0 0
[junit4:junit4] 2> 3649571 T7051 C9455 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[v!21 (1437947250389024768)]} 0 6
[junit4:junit4] 2> 3649576 T7030 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[w!22 (1437947250399510528)]} 0 1
[junit4:junit4] 2> 3649587 T7080 C9458 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[w!22 (1437947250407899136)]} 0 1
[junit4:junit4] 2> 3649590 T7052 C9455 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[w!22 (1437947250407899136)]} 0 8
[junit4:junit4] 2> 3649597 T7032 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[x!23 (1437947250421530624)]} 0 1
[junit4:junit4] 2> 3649608 T7098 C9457 P53883 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[x!23 (1437947250427822080)]} 0 1
[junit4:junit4] 2> 3649611 T7063 C9456 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[x!23 (1437947250427822080)]} 0 9
[junit4:junit4] 2> 3649617 T7031 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[y!24 (1437947250441453568)]} 0 2
[junit4:junit4] 2> 3649633 T7099 C9457 P53883 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[y!24 (1437947250451939328)]} 0 1
[junit4:junit4] 2> 3649634 T7064 C9456 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[y!24 (1437947250451939328)]} 0 9
[junit4:junit4] 2> 3649636 T7048 C9455 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[y!24]} 0 15
[junit4:junit4] 2> 3649641 T7029 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[z!25 (1437947250467667968)]} 0 1
[junit4:junit4] 2> 3649659 T7082 C9458 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[z!25 (1437947250481299456)]} 0 1
[junit4:junit4] 2> 3649661 T7049 C9455 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[z!25 (1437947250481299456)]} 0 9
[junit4:junit4] 2> 3649663 T7066 C9456 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[z!25]} 0 16
[junit4:junit4] 2> 3649673 T7028 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[a!26 (1437947250501222400)]} 0 1
[junit4:junit4] 2> 3649691 T7101 C9457 P53883 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[a!26 (1437947250515902464)]} 0 1
[junit4:junit4] 2> 3649693 T7065 C9456 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[a!26 (1437947250515902464)]} 0 7
[junit4:junit4] 2> 3649695 T7050 C9455 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[a!26]} 0 13
[junit4:junit4] 2> 3649703 T7030 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[b!27 (1437947250532679680)]} 0 1
[junit4:junit4] 2> 3649714 T7083 C9458 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[b!27 (1437947250538971136)]} 0 1
[junit4:junit4] 2> 3649716 T7053 C9455 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[b!27 (1437947250538971136)]} 0 7
[junit4:junit4] 2> 3649721 T7032 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[c!28 (1437947250551554048)]} 0 1
[junit4:junit4] 2> 3649735 T7084 C9458 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[c!28 (1437947250562039808)]} 0 1
[junit4:junit4] 2> 3649737 T7051 C9455 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[c!28 (1437947250562039808)]} 0 7
[junit4:junit4] 2> 3649739 T7067 C9456 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[c!28]} 0 13
[junit4:junit4] 2> 3649744 T7031 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[d!29 (1437947250576719872)]} 0 1
[junit4:junit4] 2> 3649758 T7100 C9457 P53883 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[d!29 (1437947250586157056)]} 0 1
[junit4:junit4] 2> 3649760 T7068 C9456 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[d!29 (1437947250586157056)]} 0 8
[junit4:junit4] 2> 3649761 T7052 C9455 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[d!29]} 0 12
[junit4:junit4] 2> 3649767 T7033 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[e!30 (1437947250600837120)]} 0 1
[junit4:junit4] 2> 3649781 T7102 C9457 P53883 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[e!30 (1437947250610274304)]} 0 1
[junit4:junit4] 2> 3649783 T7063 C9456 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[e!30 (1437947250610274304)]} 0 7
[junit4:junit4] 2> 3649786 T7048 C9455 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[e!30]} 0 14
[junit4:junit4] 2> 3649791 T7029 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[f!31 (1437947250626002944)]} 0 1
[junit4:junit4] 2> 3649805 T7097 C9457 P53883 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[f!31 (1437947250635440128)]} 0 1
[junit4:junit4] 2> 3649807 T7064 C9456 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[f!31 (1437947250635440128)]} 0 7
[junit4:junit4] 2> 3649809 T7049 C9455 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[f!31]} 0 13
[junit4:junit4] 2> 3649817 T7028 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[g!32 (1437947250652217344)]} 0 1
[junit4:junit4] 2> 3649829 T7085 C9458 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[g!32 (1437947250661654528)]} 0 0
[junit4:junit4] 2> 3649831 T7053 C9455 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[g!32 (1437947250661654528)]} 0 6
[junit4:junit4] 2> 3649833 T7066 C9456 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[g!32]} 0 12
[junit4:junit4] 2> 3649848 T7030 C9454 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[h!33 (1437947250684723200)]} 0 1
[junit4:junit4] 2> ASYNC NEW_CORE C9459 name=collection1 org.apache.solr.core.SolrCore@415cfff url=http://127.0.0.1:53879/eg/h/collection1 node=127.0.0.1:53879_eg%2Fh C9459_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:53879_eg%2Fh, base_url=http://127.0.0.1:53879/eg/h}
[junit4:junit4] 2> 3649884 T7080 C9459 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[h!33 (1437947250701500416)]} 0 0
[junit4:junit4] 2> ASYNC NEW_CORE C9460 name=collection1 org.apache.solr.core.SolrCore@40aed2d url=http://127.0.0.1:53873/eg/h/collection1 node=127.0.0.1:53873_eg%2Fh C9460_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:53873_eg%2Fh, base_url=http://127.0.0.1:53873/eg/h, leader=true}
[junit4:junit4] 2> 3649886 T7051 C9460 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[h!33 (1437947250701500416)]} 0 24
[junit4:junit4] 2> ASYNC NEW_CORE C9461 name=collection1 org.apache.solr.core.SolrCore@13563135 url=http://127.0.0.1:53876/eg/h/collection1 node=127.0.0.1:53876_eg%2Fh C9461_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:53876_eg%2Fh, base_url=http://127.0.0.1:53876/eg/h, leader=true}
[junit4:junit4] 2> 3649890 T7067 C9461 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[h!33]} 0 37
[junit4:junit4] 2> ASYNC NEW_CORE C9462 name=collection1 org.apache.solr.core.SolrCore@5706ca5 url=http://127.0.0.1:53869/eg/h/collection1 node=127.0.0.1:53869_eg%2Fh C9462_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:53869_eg%2Fh, base_url=http://127.0.0.1:53869/eg/h, leader=true}
[junit4:junit4] 2> 3649899 T7032 C9462 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[i!34 (1437947250738200576)]} 0 1
[junit4:junit4] 2> 3649919 T7082 C9459 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[i!34 (1437947250753929216)]} 0 1
[junit4:junit4] 2> 3649921 T7052 C9460 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[i!34 (1437947250753929216)]} 0 9
[junit4:junit4] 2> 3649923 T7068 C9461 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[i!34]} 0 15
[junit4:junit4] 2> 3649929 T7031 C9462 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[j!35 (1437947250769657856)]} 0 1
[junit4:junit4] 2> 3649944 T7083 C9459 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[j!35 (1437947250780143616)]} 0 1
[junit4:junit4] 2> 3649947 T7048 C9460 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[j!35 (1437947250780143616)]} 0 9
[junit4:junit4] 2> 3649949 T7063 C9461 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[j!35]} 0 15
[junit4:junit4] 2> 3649956 T7033 C9462 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[k!36 (1437947250797969408)]} 0 1
[junit4:junit4] 2> 3649968 T7084 C9459 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[k!36 (1437947250805309440)]} 0 1
[junit4:junit4] 2> 3649970 T7050 C9460 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[k!36 (1437947250805309440)]} 0 8
[junit4:junit4] 2> 3649976 T7029 C9462 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[l!37 (1437947250818940928)]} 0 1
[junit4:junit4] 2> ASYNC NEW_CORE C9463 name=collection1 org.apache.solr.core.SolrCore@392439c3 url=http://127.0.0.1:53883/eg/h/collection1 node=127.0.0.1:53883_eg%2Fh C9463_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:53883_eg%2Fh, base_url=http://127.0.0.1:53883/eg/h}
[junit4:junit4] 2> 3649987 T7098 C9463 P53883 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[l!37 (1437947250825232384)]} 0 2
[junit4:junit4] 2> 3649991 T7064 C9461 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[l!37 (1437947250825232384)]} 0 10
[junit4:junit4] 2> 3650003 T7028 C9462 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[m!38 (1437947250847252480)]} 0 2
[junit4:junit4] 2> 3650023 T7101 C9463 P53883 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[m!38 (1437947250861932544)]} 0 2
[junit4:junit4] 2> 3650025 T7065 C9461 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[m!38 (1437947250861932544)]} 0 9
[junit4:junit4] 2> 3650027 T7049 C9460 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[m!38]} 0 15
[junit4:junit4] 2> 3650034 T7030 C9462 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[n!39 (1437947250879758336)]} 0 1
[junit4:junit4] 2> 3650051 T7081 C9459 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[n!39 (1437947250892341248)]} 0 1
[junit4:junit4] 2> 3650053 T7053 C9460 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[n!39 (1437947250892341248)]} 0 8
[junit4:junit4] 2> 3650055 T7066 C9461 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[n!39]} 0 14
[junit4:junit4] 2> 3650061 T7031 C9462 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[o!40 (1437947250908069888)]} 0 1
[junit4:junit4] 2> 3650083 T7100 C9463 P53883 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[o!40 (1437947250922749952)]} 0 1
[junit4:junit4] 2> 3650085 T7067 C9461 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[o!40 (1437947250922749952)]} 0 12
[junit4:junit4] 2> 3650087 T7051 C9460 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[o!40]} 0 19
[junit4:junit4] 2> 3650103 T7033 C9462 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[p!41 (1437947250953158656)]} 0 1
[junit4:junit4] 2> 3650118 T7085 C9459 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[p!41 (1437947250962595840)]} 0 1
[junit4:junit4] 2> 3650120 T7052 C9460 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[p!41 (1437947250962595840)]} 0 8
[junit4:junit4] 2> 3650126 T7029 C9462 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[q!42 (1437947250976227328)]} 0 1
[junit4:junit4] 2> 3650139 T7080 C9459 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[q!42 (1437947250986713088)]} 0 1
[junit4:junit4] 2> 3650143 T7048 C9460 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[q!42 (1437947250986713088)]} 0 9
[junit4:junit4] 2> 3650145 T7068 C9461 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[q!42]} 0 14
[junit4:junit4] 2> 3650154 T7028 C9462 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[r!43 (1437947251005587456)]} 0 1
[junit4:junit4] 2> 3650168 T7102 C9463 P53883 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[r!43 (1437947251015024640)]} 0 1
[junit4:junit4] 2> 3650170 T7063 C9461 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[r!43 (1437947251015024640)]} 0 8
[junit4:junit4] 2> 3650172 T7050 C9460 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[r!43]} 0 14
[junit4:junit4] 2> 3650180 T7032 C9462 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[s!44 (1437947251032850432)]} 0 1
[junit4:junit4] 2> 3650198 T7082 C9459 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[s!44 (1437947251047530496)]} 0 0
[junit4:junit4] 2> 3650200 T7049 C9460 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[s!44 (1437947251047530496)]} 0 8
[junit4:junit4] 2> 3650202 T7064 C9461 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[s!44]} 0 17
[junit4:junit4] 2> 3650207 T7030 C9462 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[t!45 (1437947251061161984)]} 0 1
[junit4:junit4] 2> 3650215 T7083 C9459 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[t!45 (1437947251065356288)]} 0 1
[junit4:junit4] 2> 3650216 T7051 C9460 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[t!45 (1437947251065356288)]} 0 6
[junit4:junit4] 2> 3650222 T7031 C9462 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[u!46 (1437947251076890624)]} 0 1
[junit4:junit4] 2> 3650231 T7097 C9463 P53883 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[u!46 (1437947251082133504)]} 0 1
[junit4:junit4] 2> 3650232 T7065 C9461 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[u!46 (1437947251082133504)]} 0 6
[junit4:junit4] 2> 3650241 T7033 C9462 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[v!47 (1437947251096813568)]} 0 1
[junit4:junit4] 2> 3650253 T7084 C9459 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[v!47 (1437947251105202176)]} 0 1
[junit4:junit4] 2> 3650255 T7052 C9460 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[v!47 (1437947251105202176)]} 0 7
[junit4:junit4] 2> 3650256 T7067 C9461 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[v!47]} 0 11
[junit4:junit4] 2> 3650263 T7029 C9462 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[w!48 (1437947251119882240)]} 0 1
[junit4:junit4] 2> 3650275 T7085 C9459 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[w!48 (1437947251128270848)]} 0 1
[junit4:junit4] 2> 3650276 T7048 C9460 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[w!48 (1437947251128270848)]} 0 6
[junit4:junit4] 2> 3650278 T7068 C9461 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[w!48]} 0 11
[junit4:junit4] 2> 3650282 T7028 C9462 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[x!49 (1437947251140853760)]} 0 1
[junit4:junit4] 2> 3650298 T7099 C9463 P53883 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[x!49 (1437947251151339520)]} 0 1
[junit4:junit4] 2> 3650300 T7063 C9461 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[x!49 (1437947251151339520)]} 0 9
[junit4:junit4] 2> 3650306 T7032 C9462 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[y!50 (1437947251166019584)]} 0 1
[junit4:junit4] 2> 3650323 T7098 C9463 P53883 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[y!50 (1437947251176505344)]} 0 1
[junit4:junit4] 2> 3650325 T7064 C9461 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[y!50 (1437947251176505344)]} 0 9
[junit4:junit4] 2> 3650326 T7050 C9460 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[y!50]} 0 14
[junit4:junit4] 2> 3650333 T7030 C9462 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[z!51 (1437947251193282560)]} 0 1
[junit4:junit4] 2> 3650351 T7080 C9459 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[z!51 (1437947251209011200)]} 0 0
[junit4:junit4] 2> 3650353 T7049 C9460 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[z!51 (1437947251209011200)]} 0 7
[junit4:junit4] 2> 3650354 T7066 C9461 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[z!51]} 0 12
[junit4:junit4] 2> 3650360 T7031 C9462 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[a!52 (1437947251221594112)]} 0 1
[junit4:junit4] 2> 3650374 T7101 C9463 P53883 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[a!52 (1437947251232079872)]} 0 1
[junit4:junit4] 2> 3650376 T7065 C9461 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[a!52 (1437947251232079872)]} 0 7
[junit4:junit4] 2> 3650378 T7053 C9460 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[a!52]} 0 13
[junit4:junit4] 2> 3650385 T7033 C9462 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[b!53 (1437947251247808512)]} 0 2
[junit4:junit4] 2> 3650398 T7082 C9459 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[b!53 (1437947251256197120)]} 0 1
[junit4:junit4] 2> 3650400 T7051 C9460 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[b!53 (1437947251256197120)]} 0 9
[junit4:junit4] 2> 3650406 T7029 C9462 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[c!54 (1437947251270877184)]} 0 1
[junit4:junit4] 2> 3650416 T7083 C9459 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[c!54 (1437947251276120064)]} 0 1
[junit4:junit4] 2> 3650417 T7052 C9460 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[c!54 (1437947251276120064)]} 0 6
[junit4:junit4] 2> 3650422 T7028 C9462 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[d!55 (1437947251287654400)]} 0 1
[junit4:junit4] 2> 3650435 T7100 C9463 P53883 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[d!55 (1437947251293945856)]} 0 1
[junit4:junit4] 2> 3650439 T7067 C9461 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[d!55 (1437947251293945856)]} 0 12
[junit4:junit4] 2> 3650445 T7032 C9462 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[e!56 (1437947251310723072)]} 0 1
[junit4:junit4] 2> 3650454 T7097 C9463 P53883 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[e!56 (1437947251317014528)]} 0 1
[junit4:junit4] 2> 3650456 T7068 C9461 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[e!56 (1437947251317014528)]} 0 7
[junit4:junit4] 2> 3650462 T7031 C9462 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[f!57 (1437947251328548864)]} 0 2
[junit4:junit4] 2> 3650477 T7099 C9463 P53883 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[f!57 (1437947251340083200)]} 0 1
[junit4:junit4] 2> 3650480 T7063 C9461 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[f!57 (1437947251340083200)]} 0 8
[junit4:junit4] 2> 3650483 T7048 C9460 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[f!57]} 0 15
[junit4:junit4] 2> 3650490 T7033 C9462 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[g!58 (1437947251357908992)]} 0 2
[junit4:junit4] 2> 3650504 T7081 C9459 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[g!58 (1437947251368394752)]} 0 2
[junit4:junit4] 2> 3650506 T7050 C9460 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[g!58 (1437947251368394752)]} 0 8
[junit4:junit4] 2> 3650508 T7064 C9461 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[g!58]} 0 13
[junit4:junit4] 2> 3650514 T7029 C9462 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[h!59 (1437947251383074816)]} 0 1
[junit4:junit4] 2> 3650523 T7084 C9459 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[h!59 (1437947251389366272)]} 0 1
[junit4:junit4] 2> 3650525 T7049 C9460 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[h!59 (1437947251389366272)]} 0 6
[junit4:junit4] 2> 3650529 T7028 C9462 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[i!60 (1437947251399852032)]} 0 0
[junit4:junit4] 2> 3650547 T7085 C9459 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[i!60 (1437947251413483520)]} 0 1
[junit4:junit4] 2> 3650548 T7053 C9460 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[i!60 (1437947251413483520)]} 0 6
[junit4:junit4] 2> 3650550 T7066 C9461 P53876 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[i!60]} 0 15
[junit4:junit4] 2> 3650554 T7030 C9462 P53869 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={wt=javabin&version=2} {add=[j!61 (1437947251425017856)]} 0 1
[junit4:junit4] 2> 3650568 T7080 C9459 P53879 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53873/eg/h/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[j!61 (1437947251433406464)]} 0 1
[junit4:junit4] 2> 3650569 T7051 C9460 P53873 oasup.LogUpdateProcessor.finish [collection1] webapp=/eg/h path=/update params={distrib.from=http://127.0.0.1:53876/eg/h/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[j!61 (1437947251433406464)]} 0 8
[junit4:junit4]
[...truncated too long message...]
eper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 3693649 T7015 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/eg/h,null}
[junit4:junit4] 2> 3693729 T7015 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 3693740 T7015 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:53866 53866
[junit4:junit4] 2> 3693846 T7015 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 53869
[junit4:junit4] 2> !!!! WARNING: best effort to remove /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-1371333323951 FAILED !!!!!
[junit4:junit4] 2> 3693847 T7015 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 3693848 T7015 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:53866 53866
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ShardSplitTest -Dtests.method=testDistribSearch -Dtests.seed=5D1E545638DF3932 -Dtests.slow=true -Dtests.locale=uk -Dtests.timezone=America/Denver -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] ERROR 81.4s | ShardSplitTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:53873/eg/h returned non ok status:500, message:Server Error
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([5D1E545638DF3932:DCF8DA4E4F80590E]:0)
[junit4:junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
[junit4:junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4:junit4] > at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:228)
[junit4:junit4] > at org.apache.solr.cloud.ShardSplitTest.doTest(ShardSplitTest.java:150)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:680)
[junit4:junit4] 2> 3693905 T7015 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 81447 T7014 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4:junit4] 2> NOTE: test params are: codec=Appending, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=uk, timezone=America/Denver
[junit4:junit4] 2> NOTE: Mac OS X 10.8.4 x86_64/Apple Inc. 1.6.0_45 (64-bit)/cpus=2,threads=5,free=271263920,total=532545536
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestLuceneMatchVersion, TestCoreDiscovery, TestZkChroot, BinaryUpdateRequestHandlerTest, FullSolrCloudDistribCmdsTest, CacheHeaderTest, DebugComponentTest, EchoParamsTest, URLClassifyProcessorTest, TestFieldTypeResource, DirectUpdateHandlerOptimizeTest, TestFastWriter, TestJoin, LukeRequestHandlerTest, TestCodecSupport, MultiTermTest, SuggesterTest, DocumentBuilderTest, TestPseudoReturnFields, SynonymTokenizerTest, SliceStateTest, TestUtils, TestOmitPositions, ResourceLoaderTest, TestWriterPerf, TestConfig, TestClassNameShortening, TimeZoneUtilsTest, TestPerFieldSimilarity, AliasIntegrationTest, LeaderElectionTest, ScriptEngineTest, SpatialFilterTest, TestMergePolicyConfig, TestFastLRUCache, SuggesterWFSTTest, SignatureUpdateProcessorFactoryTest, TestSerializedLuceneMatchVersion, StatelessScriptUpdateProcessorFactoryTest, HighlighterConfigTest, PeerSyncTest, FieldAnalysisRequestHandlerTest, SolrCmdDistributorTest, UniqFieldsUpdateProcessorFactoryTest, TestCloudManagedSchema, SolrIndexSplitterTest, TestSuggestSpellingConverter, TestQueryUtils, TestPropInject, TestPhraseSuggestions, TestDocumentBuilder, TestGroupingSearch, TestSolrIndexConfig, TestCoreContainer, IndexReaderFactoryTest, DocValuesMultiTest, CoreAdminHandlerTest, CurrencyFieldOpenExchangeTest, HardAutoCommitTest, TestSolr4Spatial, SpellPossibilityIteratorTest, TestValueSourceCache, FileUtilsTest, BasicZkTest, ClusterStateUpdateTest, DisMaxRequestHandlerTest, CoreContainerCoreInitFailuresTest, TestStressVersions, RequiredFieldsTest, BasicDistributedZk2Test, IndexSchemaRuntimeFieldTest, ChaosMonkeyShardSplitTest, TestSort, TestUpdate, TestBinaryResponseWriter, CSVRequestHandlerTest, TestLRUCache, ShowFileRequestHandlerTest, TestDynamicFieldResource, TestIBSimilarityFactory, TestRandomDVFaceting, CircularListTest, QueryResultKeyTest, DateMathParserTest, TestMultiCoreConfBootstrap, TestLFUCache, DistributedQueryElevationComponentTest, SpellingQueryConverterTest, DocumentAnalysisRequestHandlerTest, DistributedTermsComponentTest, TestBinaryField, TestReversedWildcardFilterFactory, TestFieldResource, TestDefaultSimilarityFactory, MinimalSchemaTest, OpenExchangeRatesOrgProviderTest, SolrIndexConfigTest, TestReplicationHandler, HighlighterTest, TestFoldingMultitermQuery, TestRTGBase, TermVectorComponentDistributedTest, TestSolrCoreProperties, SolrInfoMBeanTest, SchemaVersionSpecificBehaviorTest, JSONWriterTest, TestSolrXMLSerializer, QueryElevationComponentTest, NoCacheHeaderTest, BadComponentTest, OutputWriterTest, RegexBoostProcessorTest, SolrPluginUtilsTest, MBeansHandlerTest, StatsComponentTest, PrimUtilsTest, OverseerCollectionProcessorTest, TestAtomicUpdateErrorCases, TestLazyCores, TestSchemaSimilarityResource, TestArbitraryIndexDir, SearchHandlerTest, XmlUpdateRequestHandlerTest, TestJmxMonitoredMap, NotRequiredUniqueKeyTest, TestDistributedSearch, ZkSolrClientTest, ZkControllerTest, MoreLikeThisHandlerTest, SuggesterFSTTest, UpdateParamsTest, TestFieldCollectionResource, AutoCommitTest, TestWordDelimiterFilterFactory, TestFieldTypeCollectionResource, TestStressLucene, QueryEqualityTest, TestPostingsSolrHighlighter, TestFunctionQuery, BadCopyFieldTest, TestSurroundQueryParser, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest, SyncSliceTest, OpenCloseCoreStressTest, OverseerTest, LeaderElectionIntegrationTest, ShardRoutingTest, RecoveryZkTest, TestRandomFaceting, ZkCLITest, ShardRoutingCustomTest, SliceStateUpdateTest, TestDistributedGrouping, TestFaceting, TestRecovery, TestHashPartitioner, DistributedSpellCheckComponentTest, TestRealTimeGet, TestStressReorder, TestReload, TestRangeQuery, SimpleFacetsTest, SolrCoreTest, SpellCheckComponentTest, BadIndexSchemaTest, ConvertedLegacyTest, TestFiltering, BasicFunctionalityTest, DirectUpdateHandlerTest, TestBadConfig, TestIndexSearcher, SoftAutoCommitTest, CurrencyFieldXmlFileTest, SimplePostToolTest, AnalysisAfterCoreReloadTest, TestExtendedDismaxParser, SpellCheckCollatorTest, SolrRequestParserTest, DocValuesTest, SuggesterTSTTest, TestTrie, TestCSVLoader, PolyFieldTest, WordBreakSolrSpellCheckerTest, SolrCoreCheckLockOnStartupTest, FieldMutatingUpdateProcessorTest, DefaultValueUpdateProcessorTest, SortByFunctionTest, TestRemoteStreaming, DistanceFunctionTest, TestSolrDeletionPolicy1, XsltUpdateRequestHandlerTest, IndexBasedSpellCheckerTest, StandardRequestHandlerTest, DirectSolrSpellCheckerTest, TestQueryTypes, PrimitiveFieldTypeTest, FileBasedSpellCheckerTest, TermsComponentTest, RequestHandlersTest, PathHierarchyTokenizerFactoryTest, TermVectorComponentTest, TestIndexingPerformance, TestSolrQueryParser, FastVectorHighlighterTest, LoggingHandlerTest, IndexSchemaTest, TestJmxIntegration, TestCollationField, ReturnFieldsTest, UpdateRequestProcessorFactoryTest, TestCSVResponseWriter, QueryParsingTest, TestAnalyzedSuggestions, JsonLoaderTest, TestPartialUpdateDeduplication, TestPHPSerializedResponseWriter, PingRequestHandlerTest, TestComponentsName, SOLR749Test, TestQuerySenderListener, TestPropInjectDefaults, AlternateDirectoryTest, TestQuerySenderNoQuery, CopyFieldTest, ResponseLogComponentTest, TestStressRecovery, TestSolrDeletionPolicy2, SampleTest, TestDocSet, TestSearchPerf, NumericFieldsTest, TestElisionMultitermQuery, TestFuzzyAnalyzedSuggestions, ExternalFileFieldSortTest, DirectSolrConnectionTest, TestCharFilters, TestXIncludeConfig, TestSweetSpotSimilarityFactory, TestDFRSimilarityFactory, TestNumberUtils, TestLMDirichletSimilarityFactory, TestLMJelinekMercerSimilarityFactory, TestBM25SimilarityFactory, LegacyHTMLStripCharFilterTest, TestPluginEnable, TestFastOutputStream, PluginInfoTest, ChaosMonkeyNothingIsSafeTest, PreAnalyzedFieldTest, TestSystemIdResolver, DateFieldTest, DOMUtilTest, RAMDirectoryFactoryTest, ClusterStateTest, TestSolrJ, ZkNodePropsTest, SystemInfoHandlerTest, UUIDFieldTest, SolrTestCaseJ4Test, TestCollationKeyRangeQueries, ShardSplitTest]
[junit4:junit4] Completed in 82.29s, 1 test, 1 error <<< FAILURES!
[...truncated 67 lines...]
BUILD FAILED
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/build.xml:392: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/build.xml:372: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build.xml:181: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/common-build.xml:437: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:1246: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:890: There were test failures: 297 suites, 1246 tests, 1 error, 18 ignored (6 assumptions)
Total time: 118 minutes 21 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.6.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-4.x-MacOSX (64bit/jdk1.7.0) - Build # 548 -
Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/548/
Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseParallelGC
1 tests failed.
REGRESSION: org.apache.solr.cloud.RecoveryZkTest.testDistribSearch
Error Message:
expected:<198> but was:<52>
Stack Trace:
java.lang.AssertionError: expected:<198> but was:<52>
at __randomizedtesting.SeedInfo.seed([97D5D962CEE15B4C:1633577AB9BE3B70]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:108)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 9701 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
[junit4:junit4] 2> 2477771 T5279 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4] 2> 2477786 T5279 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solrtest-RecoveryZkTest-1371304659327
[junit4:junit4] 2> 2477788 T5279 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 2477790 T5280 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 2477890 T5279 oasc.ZkTestServer.run start zk server on port:53249
[junit4:junit4] 2> 2477893 T5279 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2477903 T5286 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6cfcf7d3 name:ZooKeeperConnection Watcher:127.0.0.1:53249 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2477904 T5279 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2477905 T5279 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 2477919 T5279 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2477928 T5288 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c1ce1ae name:ZooKeeperConnection Watcher:127.0.0.1:53249/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2477929 T5279 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2477929 T5279 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 2477940 T5279 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 2477957 T5279 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 2477975 T5279 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 2477987 T5279 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 2477988 T5279 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 2478004 T5279 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 2478006 T5279 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 2478017 T5279 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 2478019 T5279 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 2478033 T5279 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 2478035 T5279 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 2478045 T5279 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 2478047 T5279 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 2478056 T5279 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 2478058 T5279 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 2478078 T5279 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 2478080 T5279 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 2478091 T5279 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 2478093 T5279 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 2478105 T5279 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 2478107 T5279 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 2478804 T5279 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 2478827 T5279 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53252
[junit4:junit4] 2> 2478829 T5279 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 2478829 T5279 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 2478830 T5279 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371304659660
[junit4:junit4] 2> 2478830 T5279 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371304659660/solr.xml
[junit4:junit4] 2> 2478831 T5279 oasc.CoreContainer.<init> New CoreContainer 2110545169
[junit4:junit4] 2> 2478832 T5279 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371304659660/'
[junit4:junit4] 2> 2478833 T5279 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371304659660/'
[junit4:junit4] 2> 2479062 T5279 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 2479064 T5279 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 2479064 T5279 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 2479065 T5279 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 2479066 T5279 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 2479067 T5279 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 2479067 T5279 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 2479068 T5279 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 2479069 T5279 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 2479069 T5279 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 2479105 T5279 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 2479107 T5279 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53249/solr
[junit4:junit4] 2> 2479108 T5279 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 2479119 T5279 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2479154 T5299 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12a29ce name:ZooKeeperConnection Watcher:127.0.0.1:53249 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2479159 T5279 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2479194 T5279 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 2479242 T5279 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2479245 T5301 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@212f0859 name:ZooKeeperConnection Watcher:127.0.0.1:53249/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2479246 T5279 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2479255 T5279 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 2479272 T5279 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 2479315 T5279 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 2479328 T5279 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53252_
[junit4:junit4] 2> 2479336 T5279 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53252_
[junit4:junit4] 2> 2479350 T5279 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 2479367 T5279 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 2479374 T5279 oasc.Overseer.start Overseer (id=89869822154440707-127.0.0.1:53252_-n_0000000000) starting
[junit4:junit4] 2> 2479382 T5279 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 2479394 T5303 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 2479398 T5279 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 2479408 T5279 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 2479415 T5279 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 2479434 T5302 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 2479443 T5304 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 2479444 T5304 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 2480949 T5302 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2480956 T5302 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53252_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53252"}
[junit4:junit4] 2> 2480957 T5302 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 2480957 T5302 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 2480973 T5301 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 2481452 T5304 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371304659660/collection1
[junit4:junit4] 2> 2481453 T5304 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 2481455 T5304 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 2481456 T5304 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 2481460 T5304 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371304659660/collection1/'
[junit4:junit4] 2> 2481465 T5304 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1371304659660/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 2481466 T5304 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1371304659660/collection1/lib/README' to classloader
[junit4:junit4] 2> 2481598 T5304 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 2481753 T5304 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 2481756 T5304 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 2481766 T5304 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 2483033 T5304 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 2483050 T5304 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 2483057 T5304 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 2483110 T5304 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 2483119 T5304 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 2483132 T5304 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 2483137 T5304 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 2483137 T5304 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 2483138 T5304 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 2483143 T5304 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 2483143 T5304 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 2483144 T5304 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 2483144 T5304 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371304659660/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1371304659326/control/data/
[junit4:junit4] 2> 2483145 T5304 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3652ac26
[junit4:junit4] 2> 2483146 T5304 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 2483147 T5304 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371304659326/control/data
[junit4:junit4] 2> 2483147 T5304 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1371304659326/control/data/index/
[junit4:junit4] 2> 2483148 T5304 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1371304659326/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 2483148 T5304 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371304659326/control/data/index
[junit4:junit4] 2> 2483156 T5304 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7cd0c1bc lockFactory=org.apache.lucene.store.NativeFSLockFactory@6ad234aa),segFN=segments_1,generation=1}
[junit4:junit4] 2> 2483156 T5304 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 2483161 T5304 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 2483164 T5304 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 2483175 T5304 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 2483176 T5304 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 2483177 T5304 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 2483178 T5304 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 2483178 T5304 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 2483179 T5304 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 2483180 T5304 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 2483203 T5304 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 2483216 T5304 oass.SolrIndexSearcher.<init> Opening Searcher@5a5df8e3 main
[junit4:junit4] 2> 2483217 T5304 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1371304659326/control/data/tlog
[junit4:junit4] 2> 2483219 T5304 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 2483219 T5304 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 2483224 T5305 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5a5df8e3 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 2483229 T5304 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 2483229 T5304 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53252 collection:control_collection shard:shard1
[junit4:junit4] 2> 2483233 T5304 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 2483269 T5304 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 2483282 T5304 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 2483282 T5304 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 2483283 T5304 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53252/collection1/
[junit4:junit4] 2> 2483283 T5304 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 2483284 T5304 oasc.SyncStrategy.syncToMe http://127.0.0.1:53252/collection1/ has no replicas
[junit4:junit4] 2> 2483285 T5304 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53252/collection1/
[junit4:junit4] 2> 2483285 T5304 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 2484007 T5302 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2484067 T5301 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 2484138 T5304 oasc.ZkController.register We are http://127.0.0.1:53252/collection1/ and leader is http://127.0.0.1:53252/collection1/
[junit4:junit4] 2> 2484139 T5304 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53252
[junit4:junit4] 2> 2484139 T5304 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 2484140 T5304 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 2484140 T5304 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 2484146 T5304 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2484151 T5279 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4] 2> 2484153 T5279 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 2484153 T5279 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 2484184 T5279 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 2484187 T5279 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2484196 T5308 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a72999f name:ZooKeeperConnection Watcher:127.0.0.1:53249/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2484196 T5279 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2484205 T5279 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 2484215 T5279 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 2484944 T5279 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 2484954 T5279 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53256
[junit4:junit4] 2> 2484956 T5279 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 2484957 T5279 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 2484957 T5279 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371304665755
[junit4:junit4] 2> 2484958 T5279 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371304665755/solr.xml
[junit4:junit4] 2> 2484959 T5279 oasc.CoreContainer.<init> New CoreContainer 652982270
[junit4:junit4] 2> 2484960 T5279 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371304665755/'
[junit4:junit4] 2> 2484960 T5279 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371304665755/'
[junit4:junit4] 2> 2485166 T5279 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 2485167 T5279 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 2485168 T5279 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 2485168 T5279 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 2485170 T5279 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 2485171 T5279 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 2485172 T5279 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 2485172 T5279 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 2485173 T5279 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 2485174 T5279 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 2485211 T5279 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 2485212 T5279 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53249/solr
[junit4:junit4] 2> 2485213 T5279 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 2485215 T5279 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2485620 T5319 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@35881d7c name:ZooKeeperConnection Watcher:127.0.0.1:53249 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2485622 T5279 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2485628 T5302 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2485631 T5302 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:53252__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53252_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53252"}
[junit4:junit4] 2> 2485638 T5279 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 2485656 T5301 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 2485657 T5308 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 2485693 T5279 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2485701 T5321 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8f8fe78 name:ZooKeeperConnection Watcher:127.0.0.1:53249/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2485702 T5279 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2485715 T5279 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 2486728 T5279 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53256_
[junit4:junit4] 2> 2486731 T5279 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53256_
[junit4:junit4] 2> 2486738 T5301 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 2486746 T5301 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 2486750 T5308 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 2486750 T5308 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2486751 T5321 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 2486769 T5322 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 2486770 T5322 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 2487196 T5302 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2487198 T5302 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53256_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53256"}
[junit4:junit4] 2> 2487199 T5302 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4] 2> 2487199 T5302 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 2487212 T5301 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2487212 T5321 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2487212 T5308 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2487777 T5322 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371304665755/collection1
[junit4:junit4] 2> 2487778 T5322 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 2487779 T5322 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 2487780 T5322 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 2487783 T5322 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371304665755/collection1/'
[junit4:junit4] 2> 2487785 T5322 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1371304665755/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 2487786 T5322 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1371304665755/collection1/lib/README' to classloader
[junit4:junit4] 2> 2487908 T5322 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 2488033 T5322 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 2488036 T5322 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 2488044 T5322 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 2489640 T5322 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 2489695 T5322 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 2489702 T5322 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 2489746 T5322 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 2489755 T5322 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 2489765 T5322 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 2489769 T5322 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 2489771 T5322 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 2489772 T5322 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 2489774 T5322 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 2489775 T5322 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 2489775 T5322 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 2489776 T5322 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371304665755/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1371304659326/jetty1/
[junit4:junit4] 2> 2489776 T5322 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3652ac26
[junit4:junit4] 2> 2489777 T5322 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 2489778 T5322 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371304659326/jetty1
[junit4:junit4] 2> 2489778 T5322 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1371304659326/jetty1/index/
[junit4:junit4] 2> 2489778 T5322 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1371304659326/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 2489779 T5322 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371304659326/jetty1/index
[junit4:junit4] 2> 2489786 T5322 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@a167e0c lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bf4407c),segFN=segments_1,generation=1}
[junit4:junit4] 2> 2489787 T5322 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 2489799 T5322 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 2489800 T5322 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 2489801 T5322 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 2489802 T5322 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 2489803 T5322 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 2489803 T5322 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 2489804 T5322 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 2489806 T5322 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 2489807 T5322 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 2489840 T5322 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 2489860 T5322 oass.SolrIndexSearcher.<init> Opening Searcher@4a7668c main
[junit4:junit4] 2> 2489861 T5322 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1371304659326/jetty1/tlog
[junit4:junit4] 2> 2489863 T5322 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 2489863 T5322 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 2489871 T5323 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4a7668c main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 2489877 T5322 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 2489877 T5322 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53256 collection:collection1 shard:shard1
[junit4:junit4] 2> 2489880 T5322 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 2489915 T5322 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 2489924 T5322 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 2489925 T5322 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 2489926 T5322 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53256/collection1/
[junit4:junit4] 2> 2489926 T5322 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 2489927 T5322 oasc.SyncStrategy.syncToMe http://127.0.0.1:53256/collection1/ has no replicas
[junit4:junit4] 2> 2489927 T5322 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53256/collection1/
[junit4:junit4] 2> 2489928 T5322 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 2490238 T5302 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2490260 T5301 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2490260 T5308 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2490260 T5321 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2490270 T5322 oasc.ZkController.register We are http://127.0.0.1:53256/collection1/ and leader is http://127.0.0.1:53256/collection1/
[junit4:junit4] 2> 2490271 T5322 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53256
[junit4:junit4] 2> 2490272 T5322 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 2490272 T5322 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 2490273 T5322 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 2490278 T5322 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2490282 T5279 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4] 2> 2490282 T5279 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 2490283 T5279 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 2490996 T5279 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 2491004 T5279 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53259
[junit4:junit4] 2> 2491005 T5279 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 2491006 T5279 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 2491007 T5279 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371304671870
[junit4:junit4] 2> 2491007 T5279 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371304671870/solr.xml
[junit4:junit4] 2> 2491008 T5279 oasc.CoreContainer.<init> New CoreContainer 2010952830
[junit4:junit4] 2> 2491010 T5279 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371304671870/'
[junit4:junit4] 2> 2491011 T5279 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371304671870/'
[junit4:junit4] 2> 2491233 T5279 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 2491234 T5279 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 2491235 T5279 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 2491235 T5279 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 2491236 T5279 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 2491236 T5279 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 2491237 T5279 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 2491237 T5279 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 2491238 T5279 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 2491238 T5279 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 2491267 T5279 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 2491268 T5279 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53249/solr
[junit4:junit4] 2> 2491269 T5279 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 2491272 T5279 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2491285 T5335 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1aa61bd4 name:ZooKeeperConnection Watcher:127.0.0.1:53249 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2491287 T5279 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2491291 T5279 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 2491316 T5279 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2491322 T5337 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@78347137 name:ZooKeeperConnection Watcher:127.0.0.1:53249/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2491323 T5279 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2491332 T5279 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 2491782 T5302 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2491784 T5302 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:53256__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53256_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53256"}
[junit4:junit4] 2> 2491799 T5337 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2491800 T5321 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2491802 T5301 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2491802 T5308 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2492342 T5279 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53259_
[junit4:junit4] 2> 2492345 T5279 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53259_
[junit4:junit4] 2> 2492351 T5321 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2492355 T5301 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2492355 T5308 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 2492358 T5308 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2492355 T5337 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 2492361 T5337 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2492370 T5321 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 2492375 T5301 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 2492387 T5338 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 2492387 T5338 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 2493324 T5302 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2493327 T5302 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53259_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53259"}
[junit4:junit4] 2> 2493327 T5302 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 2493328 T5302 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 2493339 T5301 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2493340 T5337 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2493341 T5308 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2493340 T5321 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2493391 T5338 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371304671870/collection1
[junit4:junit4] 2> 2493392 T5338 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 2493394 T5338 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 2493394 T5338 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 2493398 T5338 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371304671870/collection1/'
[junit4:junit4] 2> 2493399 T5338 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1371304671870/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 2493400 T5338 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1371304671870/collection1/lib/README' to classloader
[junit4:junit4] 2> 2493495 T5338 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 2493601 T5338 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 2493605 T5338 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 2493617 T5338 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 2494697 T5338 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 2494712 T5338 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 2494717 T5338 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 2494764 T5338 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 2494776 T5338 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 2494785 T5338 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 2494788 T5338 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 2494790 T5338 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 2494791 T5338 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 2494795 T5338 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 2494796 T5338 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 2494797 T5338 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 2494798 T5338 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371304671870/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1371304659326/jetty2/
[junit4:junit4] 2> 2494799 T5338 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3652ac26
[junit4:junit4] 2> 2494799 T5338 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 2494801 T5338 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371304659326/jetty2
[junit4:junit4] 2> 2494801 T5338 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1371304659326/jetty2/index/
[junit4:junit4] 2> 2494802 T5338 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1371304659326/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 2494803 T5338 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371304659326/jetty2/index
[junit4:junit4] 2> 2494811 T5338 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@68baee3f lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cb1c2c4),segFN=segments_1,generation=1}
[junit4:junit4] 2> 2494811 T5338 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 2494818 T5338 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 2494818 T5338 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 2494819 T5338 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 2494820 T5338 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 2494822 T5338 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 2494823 T5338 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 2494825 T5338 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 2494829 T5338 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 2494830 T5338 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 2494860 T5338 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 2494872 T5338 oass.SolrIndexSearcher.<init> Opening Searcher@51383def main
[junit4:junit4] 2> 2494873 T5338 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1371304659326/jetty2/tlog
[junit4:junit4] 2> 2494875 T5338 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 2494876 T5338 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 2494884 T5339 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@51383def main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 2494902 T5338 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 2494904 T5338 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53259 collection:collection1 shard:shard1
[junit4:junit4] 2> 2494914 T5338 oasc.ZkController.register We are http://127.0.0.1:53259/collection1/ and leader is http://127.0.0.1:53256/collection1/
[junit4:junit4] 2> 2494914 T5338 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53259
[junit4:junit4] 2> 2494914 T5338 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 2494915 T5338 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C1214 name=collection1 org.apache.solr.core.SolrCore@7cbce64a url=http://127.0.0.1:53259/collection1 node=127.0.0.1:53259_ C1214_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:53259_, base_url=http://127.0.0.1:53259}
[junit4:junit4] 2> 2494917 T5340 C1214 P53259 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 2494918 T5340 C1214 P53259 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 2494923 T5340 C1214 P53259 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 2494923 T5340 C1214 P53259 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 2494925 T5338 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2494926 T5340 C1214 P53259 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 2494938 T5279 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4] 2> 2494939 T5279 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 2494940 T5279 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 2494970 T5279 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2494973 T5316 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:53259__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> ASYNC NEW_CORE C1215 name=collection1 org.apache.solr.core.SolrCore@6eea934e url=http://127.0.0.1:53252/collection1 node=127.0.0.1:53252_ C1215_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:53252_, base_url=http://127.0.0.1:53252, leader=true}
[junit4:junit4] 2> 2495002 T5297 C1215 P53252 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7cd0c1bc lockFactory=org.apache.lucene.store.NativeFSLockFactory@6ad234aa),segFN=segments_1,generation=1}
[junit4:junit4] 2> 2495003 T5297 C1215 P53252 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 2495015 T5296 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1 (1437917172506558464)]} 0 15
[junit4:junit4] 2> 2495018 T5297 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10001 (1437917172504461312)]} 0 20
[junit4:junit4] 2> ASYNC NEW_CORE C1216 name=collection1 org.apache.solr.core.SolrCore@26e891ef url=http://127.0.0.1:53256/collection1 node=127.0.0.1:53256_ C1216_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:53256_, base_url=http://127.0.0.1:53256, leader=true}
[junit4:junit4] 2> 2495066 T5312 C1216 P53256 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@a167e0c lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bf4407c),segFN=segments_1,generation=1}
[junit4:junit4] 2> 2495067 T5312 C1216 P53256 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 2495071 T5312 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10001 (1437917172572618752)]} 0 9
[junit4:junit4] 2> 2495071 T5313 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1 (1437917172573667328)]} 0 7
[junit4:junit4] 2> 2495103 T5293 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10002 (1437917172613513216)]} 0 1
[junit4:junit4] 2> 2495104 T5292 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2 (1437917172613513217)]} 0 2
[junit4:junit4] 2> 2495109 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2 (1437917172619804672)]} 0 1
[junit4:junit4] 2> 2495110 T5314 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10002 (1437917172619804673)]} 0 2
[junit4:junit4] 2> 2495183 T5294 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[1 (-1437917172697399296)]} 0 1
[junit4:junit4] 2> 2495183 T5295 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10001 (-1437917172697399297)]} 0 1
[junit4:junit4] 2> 2495186 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[1 (-1437917172701593600)]} 0 0
[junit4:junit4] 2> 2495187 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10001 (-1437917172702642176)]} 0 0
[junit4:junit4] 2> 2495190 T5297 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[3 (1437917172704739328)]} 0 1
[junit4:junit4] 2> 2495191 T5296 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10003 (1437917172705787904)]} 0 1
[junit4:junit4] 2> 2495195 T5313 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[3 (1437917172709982208)]} 0 1
[junit4:junit4] 2> 2495197 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10003 (1437917172712079360)]} 0 1
[junit4:junit4] 2> 2495296 T5293 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[2 (-1437917172816936960)]} 0 0
[junit4:junit4] 2> 2495297 T5294 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10002 (-1437917172817985536)]} 0 0
[junit4:junit4] 2> 2495302 T5314 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[2 (-1437917172822179840)]} 0 2
[junit4:junit4] 2> 2495302 T5312 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10002 (-1437917172823228416)]} 0 0
[junit4:junit4] 2> 2495307 T5295 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[4 (1437917172826374144)]} 0 2
[junit4:junit4] 2> 2495308 T5297 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10004 (1437917172828471296)]} 0 1
[junit4:junit4] 2> 2495313 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[4 (1437917172832665600)]} 0 2
[junit4:junit4] 2> 2495316 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10004 (1437917172836859904)]} 0 1
[junit4:junit4] 2> 2495403 T5292 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[5 (1437917172928086017)]} 0 2
[junit4:junit4] 2> 2495404 T5296 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10005 (1437917172928086016)]} 0 3
[junit4:junit4] 2> 2495410 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[5 (1437917172934377472)]} 0 2
[junit4:junit4] 2> 2495411 T5314 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10005 (1437917172935426048)]} 0 2
[junit4:junit4] 2> 2495438 T5293 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[6 (1437917172964786176)]} 0 2
[junit4:junit4] 2> 2495442 T5294 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10006 (1437917172965834752)]} 0 4
[junit4:junit4] 2> 2495448 T5312 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10006 (1437917172975271936)]} 0 1
[junit4:junit4] 2> 2495451 T5312 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[6 (1437917172978417664)]} 0 2
[junit4:junit4] 2> 2495460 T5295 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10007 (1437917172987854848)]} 0 1
[junit4:junit4] 2> 2495463 T5295 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[7 (1437917172991000576)]} 0 1
[junit4:junit4] 2> 2495469 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10007 (1437917172997292032)]} 0 1
[junit4:junit4] 2> 2495471 T5314 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[7 (1437917172997292033)]} 0 3
[junit4:junit4] 2> 2495524 T5292 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10008 (1437917173054963712)]} 0 2
[junit4:junit4] 2> 2495530 T5296 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[8 (1437917173060206592)]} 0 2
[junit4:junit4] 2> 2495534 T5312 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10008 (1437917173064400896)]} 0 2
[junit4:junit4] 2> 2495538 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[8 (1437917173068595200)]} 0 2
[junit4:junit4] 2> 2495618 T5293 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10005 (-1437917173154578432)]} 0 1
[junit4:junit4] 2> 2495627 T5294 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[5 (-1437917173162967040)]} 0 2
[junit4:junit4] 2> 2495635 T5313 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10005 (-1437917173172404224)]} 0 1
[junit4:junit4] 2> 2495637 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[5 (-1437917173174501376)]} 0 0
[junit4:junit4] 2> 2495641 T5295 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10009 (1437917173177647104)]} 0 1
[junit4:junit4] 2> 2495646 T5297 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[9 (1437917173182889984)]} 0 1
[junit4:junit4] 2> 2495651 T5314 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10009 (1437917173188132864)]} 0 2
[junit4:junit4] 2> 2495654 T5312 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[9 (1437917173192327168)]} 0 1
[junit4:junit4] 2> 2495691 T5292 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10006 (-1437917173231124480)]} 0 1
[junit4:junit4] 2> 2495694 T5296 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[6 (-1437917173234270208)]} 0 0
[junit4:junit4] 2> 2495701 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10006 (-1437917173240561664)]} 0 1
[junit4:junit4] 2> 2495702 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[6 (-1437917173242658816)]} 0 0
[junit4:junit4] 2> 2495706 T5294 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10010 (1437917173244755968)]} 0 2
[junit4:junit4] 2> 2495708 T5294 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10 (1437917173247901696)]} 0 1
[junit4:junit4] 2> 2495712 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10 (1437917173253144576)]} 0 0
[junit4:junit4] 2> 2495716 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10010 (1437917173256290304)]} 0 1
[junit4:junit4] 2> 2495760 T5297 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[11 (1437917173300330496)]} 0 4
[junit4:junit4] 2> 2495767 T5292 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10011 (1437917173308719104)]} 0 10
[junit4:junit4] 2> 2495769 T5312 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[11 (1437917173311864832)]} 0 1
[junit4:junit4] 2> 2495770 T5313 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10011 (1437917173313961984)]} 0 0
[junit4:junit4] 2> 2495791 T5296 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[7 (-1437917173335982080)]} 0 0
[junit4:junit4] 2> 2495794 T5293 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10007 (-1437917173338079232)]} 0 1
[junit4:junit4] 2> 2495796 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[7 (-1437917173341224960)]} 0 0
[junit4:junit4] 2> 2495798 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10007 (-1437917173343322112)]} 0 0
[junit4:junit4] 2> 2495802 T5294 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[12 (1437917173346467840)]} 0 2
[junit4:junit4] 2> 2495805 T5295 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10012 (1437917173349613568)]} 0 1
[junit4:junit4] 2> 2495808 T5314 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[12 (1437917173352759296)]} 0 1
[junit4:junit4] 2> 2495814 T5312 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10012 (1437917173360099328)]} 0 1
[junit4:junit4] 2> 2495917 T5297 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[13 (1437917173466005504)]} 0 2
[junit4:junit4] 2> 2495917 T5292 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10013 (1437917173467054080)]} 0 2
[junit4:junit4] 2> 2495924 T5313 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[13 (1437917173475442688)]} 0 1
[junit4:junit4] 2> 2495927 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10013 (1437917173478588416)]} 0 0
[junit4:junit4] 2> 2495956 T5296 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[13 (-1437917173508997120)]} 0 0
[junit4:junit4] 2> 2495959 T5293 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10013 (-1437917173512142848)]} 0 0
[junit4:junit4] 2> 2495963 T5314 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[13 (-1437917173516337152)]} 0 0
[junit4:junit4] 2> 2495966 T5312 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10013 (-1437917173518434304)]} 0 1
[junit4:junit4] 2> 2495970 T5294 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[14 (1437917173522628608)]} 0 2
[junit4:junit4] 2> 2495975 T5295 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10014 (1437917173526822912)]} 0 2
[junit4:junit4] 2> 2495979 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[14 (1437917173532065792)]} 0 1
[junit4:junit4] 2> 2495982 T5313 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10014 (1437917173535211520)]} 0 1
[junit4:junit4] 2> 2496063 T5297 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[14 (-1437917173621194752)]} 0 0
[junit4:junit4] 2> 2496065 T5292 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10014 (-1437917173623291904)]} 0 0
[junit4:junit4] 2> 2496068 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[14 (-1437917173626437632)]} 0 1
[junit4:junit4] 2> 2496069 T5314 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10014 (-1437917173627486208)]} 0 0
[junit4:junit4] 2> 2496073 T5296 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[15 (1437917173629583360)]} 0 2
[junit4:junit4] 2> 2496074 T5293 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10015 (1437917173631680512)]} 0 2
[junit4:junit4] 2> 2496077 T5312 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[15 (1437917173634826240)]} 0 1
[junit4:junit4] 2> 2496077 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10015 (1437917173635874816)]} 0 0
[junit4:junit4] 2> 2496139 T5295 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10016 (1437917173699837952)]} 0 1
[junit4:junit4] 2> 2496140 T5294 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[16 (1437917173699837953)]} 0 3
[junit4:junit4] 2> 2496144 T5313 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10016 (1437917173705080832)]} 0 1
[junit4:junit4] 2> 2496147 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[16 (1437917173708226560)]} 0 1
[junit4:junit4] 2> 2496177 T5292 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10016 (-1437917173740732416)]} 0 0
[junit4:junit4] 2> 2496179 T5296 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[16 (-1437917173742829568)]} 0 1
[junit4:junit4] 2> 2496181 T5314 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10016 (-1437917173744926720)]} 0 0
[junit4:junit4] 2> 2496183 T5312 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[16 (-1437917173747023872)]} 0 0
[junit4:junit4] 2> 2496186 T5293 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10017 (1437917173749121024)]} 0 1
[junit4:junit4] 2> 2496189 T5297 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[17 (1437917173752266752)]} 0 1
[junit4:junit4] 2> 2496191 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10017 (1437917173754363904)]} 0 1
[junit4:junit4] 2> 2496192 T5313 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[17 (1437917173756461056)]} 0 0
[junit4:junit4] 2> 2496232 T5294 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[17 (-1437917173797355521)]} 0 1
[junit4:junit4] 2> 2496232 T5295 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10017 (-1437917173797355520)]} 0 1
[junit4:junit4] 2> 2496235 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[17 (-1437917173801549824)]} 0 0
[junit4:junit4] 2> 2496236 T5314 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10017 (-1437917173802598400)]} 0 0
[junit4:junit4] 2> 2496243 T5292 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[18 (1437917173805744128)]} 0 4
[junit4:junit4] 2> 2496243 T5296 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10018 (1437917173808889856)]} 0 4
[junit4:junit4] 2> 2496248 T5312 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[18 (1437917173814132736)]} 0 1
[junit4:junit4] 2> 2496251 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10018 (1437917173817278464)]} 0 1
[junit4:junit4] 2> 2496265 T5293 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[19 (1437917173831958528)]} 0 1
[junit4:junit4] 2> 2496267 T5293 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10019 (1437917173834055680)]} 0 1
[junit4:junit4] 2> 2496272 T5313 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10019 (1437917173839298560)]} 0 1
[junit4:junit4] 2> 2496272 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[19 (1437917173839298561)]} 0 1
[junit4:junit4] 2> 2496364 T5294 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10020 (1437917173935767552)]} 0 2
[junit4:junit4] 2> 2496366 T5294 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[20 (1437917173937864704)]} 0 1
[junit4:junit4] 2> 2496371 T5314 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[20 (1437917173943107584)]} 0 1
[junit4:junit4] 2> 2496372 T5312 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10020 (1437917173943107585)]} 0 2
[junit4:junit4] 2> 2496398 T5302 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2496402 T5302 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:53259__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53259_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53259"}
[junit4:junit4] 2> 2496403 T5292 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10021 (1437917173975613440)]} 0 2
[junit4:junit4] 2> 2496404 T5296 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[21 (1437917173975613441)]} 0 3
[junit4:junit4] 2> 2496409 T5313 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10021 (1437917173982953472)]} 0 1
[junit4:junit4] 2> 2496412 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[21 (1437917173986099200)]} 0 1
[junit4:junit4] 2> 2496421 T5301 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2496421 T5308 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2496421 T5337 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2496422 T5321 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2496471 T5293 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[22 (1437917174047965184)]} 0 1
[junit4:junit4] 2> 2496471 T5297 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10022 (1437917174047965185)]} 0 1
[junit4:junit4] 2> C1214_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:53259_, base_url=http://127.0.0.1:53259}
[junit4:junit4] 2> 2496517 T5329 C1214 P53259 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@68baee3f lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cb1c2c4),segFN=segments_1,generation=1}
[junit4:junit4] 2> 2496520 T5329 C1214 P53259 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 2496524 T5329 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10022 (1437917174055305216)]} 0 11
[junit4:junit4] 2> 2496524 T5330 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[22 (1437917174053208064)]} 0 8
[junit4:junit4] 2> 2496525 T5314 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10022 (1437917174055305216)]} 0 49
[junit4:junit4] 2> 2496527 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[22 (1437917174053208064)]} 0 52
[junit4:junit4] 2> 2496536 T5295 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[18 (-1437917174117171201)]} 0 0
[junit4:junit4] 2> 2496536 T5294 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10018 (-1437917174117171200)]} 0 0
[junit4:junit4] 2> 2496544 T5331 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[18 (-1437917174121365504)]} 0 0
[junit4:junit4] 2> 2496547 T5331 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10018 (-1437917174123462656)]} 0 1
[junit4:junit4] 2> 2496548 T5312 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[18 (-1437917174121365504)]} 0 8
[junit4:junit4] 2> 2496549 T5313 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10018 (-1437917174123462656)]} 0 7
[junit4:junit4] 2> 2496553 T5296 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[23 (1437917174133948416)]} 0 1
[junit4:junit4] 2> 2496556 T5293 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10023 (1437917174137094144)]} 0 1
[junit4:junit4] 2> 2496566 T5333 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[23 (1437917174141288448)]} 0 2
[junit4:junit4] 2> 2496569 T5333 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10023 (1437917174143385600)]} 0 1
[junit4:junit4] 2> 2496571 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[23 (1437917174141288448)]} 0 13
[junit4:junit4] 2> 2496572 T5314 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10023 (1437917174143385600)]} 0 11
[junit4:junit4] 2> 2496588 T5297 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10019 (-1437917174171697152)]} 0 1
[junit4:junit4] 2> 2496588 T5295 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[19 (-1437917174171697153)]} 0 0
[junit4:junit4] 2> 2496598 T5330 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10019 (-1437917174175891456)]} 0 2
[junit4:junit4] 2> 2496601 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10019 (-1437917174175891456)]} 0 9
[junit4:junit4] 2> 2496599 T5329 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[19 (-1437917174176940032)]} 0 3
[junit4:junit4] 2> 2496605 T5312 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[19 (-1437917174176940032)]} 0 13
[junit4:junit4] 2> 2496610 T5292 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10024 (1437917174192668672)]} 0 3
[junit4:junit4] 2> 2496620 T5292 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[24 (1437917174204203008)]} 0 1
[junit4:junit4] 2> 2496629 T5332 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10024 (1437917174207348736)]} 0 2
[junit4:junit4] 2> 2496629 T5331 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[24 (1437917174209445888)]} 0 2
[junit4:junit4] 2> 2496631 T5313 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10024 (1437917174207348736)]} 0 9
[junit4:junit4] 2> 2496631 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[24 (1437917174209445888)]} 0 7
[junit4:junit4] 2> 2496685 T5296 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[25 (1437917174272360448)]} 0 2
[junit4:junit4] 2> 2496685 T5293 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10025 (1437917174272360449)]} 0 1
[junit4:junit4] 2> 2496693 T5333 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[25 (1437917174276554752)]} 0 2
[junit4:junit4] 2> 2496694 T5330 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10025 (1437917174277603328)]} 0 1
[junit4:junit4] 2> 2496694 T5314 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[25 (1437917174276554752)]} 0 6
[junit4:junit4] 2> 2496696 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10025 (1437917174277603328)]} 0 7
[junit4:junit4] 2> 2496800 T5295 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[26 (1437917174392946688)]} 0 1
[junit4:junit4] 2> 2496800 T5297 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10026 (1437917174392946689)]} 0 1
[junit4:junit4] 2> 2496808 T5329 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[26 (1437917174397140992)]} 0 1
[junit4:junit4] 2> 2496809 T5332 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10026 (1437917174399238144)]} 0 1
[junit4:junit4] 2> 2496809 T5312 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[26 (1437917174397140992)]} 0 6
[junit4:junit4] 2> 2496810 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10026 (1437917174399238144)]} 0 5
[junit4:junit4] 2> 2496908 T5294 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[27 (1437917174506192896)]} 0 2
[junit4:junit4] 2> 2496908 T5292 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10027 (1437917174506192897)]} 0 1
[junit4:junit4] 2> 2496916 T5331 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[27 (1437917174510387200)]} 0 1
[junit4:junit4] 2> 2496917 T5328 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10027 (1437917174511435776)]} 0 2
[junit4:junit4] 2> 2496918 T5314 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[27 (1437917174510387200)]} 0 7
[junit4:junit4] 2> 2496920 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10027 (1437917174511435776)]} 0 8
[junit4:junit4] 2> 2496978 T5316 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:53259__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 2496979 T5316 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:53259__collection1&state=recovering&nodeName=127.0.0.1:53259_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2006
[junit4:junit4] 2> 2496985 T5279 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 53259
[junit4:junit4] 2> 2496986 T5279 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=2010952830
[junit4:junit4] 2> 2496998 T5296 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[21 (-1437917174600564736)]} 0 1
[junit4:junit4] 2> 2496998 T5295 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10021 (-1437917174601613312)]} 0 0
[junit4:junit4] 2> 2497006 T5333 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10021 (-1437917174606856192)]} 0 1
[junit4:junit4] 2> 2497006 T5330 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[21 (-1437917174606856193)]} 0 1
[junit4:junit4] 2> 2497008 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10021 (-1437917174606856192)]} 0 5
[junit4:junit4] 2> 2497008 T5312 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[21 (-1437917174606856193)]} 0 5
[junit4:junit4] 2> 2497012 T5297 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10028 (1437917174615244800)]} 0 1
[junit4:junit4] 2> 2497012 T5294 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[28 (1437917174615244801)]} 0 1
[junit4:junit4] 2> 2497027 T5329 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10028 (1437917174625730560)]} 0 1
[junit4:junit4] 2> 2497027 T5332 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[28 (1437917174626779136)]} 0 1
[junit4:junit4] 2> 2497031 T5314 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10028 (1437917174625730560)]} 0 12
[junit4:junit4] 2> 2497031 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[28 (1437917174626779136)]} 0 9
[junit4:junit4] 2> 2497064 T5292 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10029 (1437917174668722176)]} 0 2
[junit4:junit4] 2> 2497068 T5293 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[29 (1437917174672916480)]} 0 2
[junit4:junit4] 2> 2497075 T5331 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10029 (1437917174676062208)]} 0 2
[junit4:junit4] 2> 2497081 T5316 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10029 (1437917174676062208)]} 0 12
[junit4:junit4] 2> 2497084 T5328 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[29 (1437917174679207936)]} 0 1
[junit4:junit4] 2> 2497086 T5313 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[29 (1437917174679207936)]} 0 14
[junit4:junit4] 2> 2497151 T5296 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10026 (-1437917174762045440)]} 0 0
[junit4:junit4] 2> 2497152 T5295 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[26 (-1437917174763094016)]} 0 0
[junit4:junit4] 2> 2497164 T5333 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10026 (-1437917174771482624)]} 0 1
[junit4:junit4] 2> 2497165 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10026 (-1437917174771482624)]} 0 5
[junit4:junit4] 2> 2497166 T5329 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[26 (-1437917174774628352)]} 0 0
[junit4:junit4] 2> 2497167 T5312 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[26 (-1437917174774628352)]} 0 4
[junit4:junit4] 2> 2497169 T5297 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10030 (1437917174780919808)]} 0 0
[junit4:junit4] 2> 2497172 T5294 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[30 (1437917174783016960)]} 0 1
[junit4:junit4] 2> 2497178 T5332 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10030 (1437917174785114112)]} 0 1
[junit4:junit4] 2> 2497181 T5332 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[30 (1437917174788259840)]} 0 2
[junit4:junit4] 2> 2497182 T5314 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10030 (1437917174785114112)]} 0 9
[junit4:junit4] 2> 2497183 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[30 (1437917174788259840)]} 0 8
[junit4:junit4] 2> 2497274 T5292 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10031 (1437917174889971712)]} 0 1
[junit4:junit4] 2> 2497275 T5293 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[31 (1437917174891020288)]} 0 2
[junit4:junit4] 2> 2497291 T5330 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10031 (1437917174896263168)]} 0 1
[junit4:junit4] 2> 2497291 T5328 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[31 (1437917174896263169)]} 0 1
[junit4:junit4] 2> 2497292 T5316 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10031 (1437917174896263168)]} 0 14
[junit4:junit4] 2> 2497294 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[31 (1437917174896263169)]} 0 16
[junit4:junit4] 2> 2497309 T5295 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10027 (-1437917174927720448)]} 0 1
[junit4:junit4] 2> 2497309 T5296 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[27 (-1437917174927720449)]} 0 1
[junit4:junit4] 2> 2497318 T5333 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10027 (-1437917174932963328)]} 0 1
[junit4:junit4] 2> 2497320 T5312 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10027 (-1437917174932963328)]} 0 6
[junit4:junit4] 2> 2497320 T5333 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[27 (-1437917174934011904)]} 0 0
[junit4:junit4] 2> 2497331 T5314 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[27 (-1437917174934011904)]} 0 16
[junit4:junit4] 2> 2497333 T5297 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10032 (1437917174951837696)]} 0 1
[junit4:junit4] 2> 2497338 T5294 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[32 (1437917174957080576)]} 0 1
[junit4:junit4] 2> 2497342 T5332 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10032 (1437917174957080576)]} 0 1
[junit4:junit4] 2> 2497344 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10032 (1437917174957080576)]} 0 7
[junit4:junit4] 2> 2497348 T5331 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[32 (1437917174964420608)]} 0 1
[junit4:junit4] 2> 2497350 T5313 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[32 (1437917174964420608)]} 0 7
[junit4:junit4] 2> 2497422 T5292 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10031 (-1437917175046209536)]} 0 1
[junit4:junit4] 2> 2497428 T5330 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10031 (-1437917175049355264)]} 0 0
[junit4:junit4] 2> 2497431 T5316 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10031 (-1437917175049355264)]} 0 6
[junit4:junit4] 2> 2497433 T5295 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[31 (-1437917175057743872)]} 0 0
[junit4:junit4] 2> 2497436 T5296 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10033 (1437917175059841024)]} 0 1
[junit4:junit4] 2> 2497444 T5328 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[31 (-1437917175061938176)]} 0 0
[junit4:junit4] 2> 2497445 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[31 (-1437917175061938176)]} 0 8
[junit4:junit4] 2> 2497448 T5329 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10033 (1437917175068229632)]} 0 1
[junit4:junit4] 2> 2497453 T5297 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[33 (1437917175077666816)]} 0 1
[junit4:junit4] 2> 2497455 T5312 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10033 (1437917175068229632)]} 0 12
[junit4:junit4] 2> 2497464 T5333 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[33 (1437917175085006848)]} 0 1
[junit4:junit4] 2> 2497470 T5314 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[33 (1437917175085006848)]} 0 11
[junit4:junit4] 2> 2497485 T5294 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10034 (1437917175111221248)]} 0 1
[junit4:junit4] 2> 2497495 T5332 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10034 (1437917175118561280)]} 0 1
[junit4:junit4] 2> 2497496 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10034 (1437917175118561280)]} 0 6
[junit4:junit4] 2> 2497501 T5293 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[34 (1437917175127998464)]} 0 1
[junit4:junit4] 2> 2497509 T5331 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[34 (1437917175132192768)]} 0 2
[junit4:junit4] 2> 2497510 T5313 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[34 (1437917175132192768)]} 0 6
[junit4:junit4] 2> 2497512 T5292 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10035 (1437917175139532800)]} 0 1
[junit4:junit4] 2> 2497525 T5330 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10035 (1437917175150018560)]} 0 1
[junit4:junit4] 2> 2497525 T5295 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[35 (1437917175154212864)]} 0 0
[junit4:junit4] 2> 2497527 T5316 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10035 (1437917175150018560)]} 0 7
[junit4:junit4] 2> 2497536 T5328 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[35 (1437917175161552896)]} 0 1
[junit4:junit4] 2> 2497537 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[35 (1437917175161552896)]} 0 6
[junit4:junit4] 2> 2497592 T5296 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10034 (-1437917175223418880)]} 0 1
[junit4:junit4] 2> 2497603 T5329 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10034 (-1437917175227613184)]} 0 2
[junit4:junit4] 2> 2497605 T5297 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[34 (-1437917175238098944)]} 0 1
[junit4:junit4] 2> 2497605 T5312 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10034 (-1437917175227613184)]} 0 10
[junit4:junit4] 2> 2497612 T5294 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10036 (1437917175242293248)]} 0 3
[junit4:junit4] 2> 2497612 T5332 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[34 (-1437917175241244672)]} 0 0
[junit4:junit4] 2> 2497614 T5314 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[34 (-1437917175241244672)]} 0 6
[junit4:junit4] 2> 2497619 T5293 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[36 (1437917175251730432)]} 0 1
[junit4:junit4] 2> 2497621 T5331 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10036 (1437917175249633280)]} 0 1
[junit4:junit4] 2> 2497623 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10036 (1437917175249633280)]} 0 7
[junit4:junit4] 2> 2497625 T5330 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[36 (1437917175255924736)]} 0 0
[junit4:junit4] 2> 2497627 T5313 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[36 (1437917175255924736)]} 0 5
[junit4:junit4] 2> 2497712 T5292 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[37 (1437917175349248000)]} 0 1
[junit4:junit4] 2> 2497715 T5292 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10037 (1437917175352393728)]} 0 1
[junit4:junit4] 2> 2497724 T5328 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[37 (1437917175356588032)]} 0 1
[junit4:junit4] 2> 2497728 T5333 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10037 (1437917175358685184)]} 0 2
[junit4:junit4] 2> 2497729 T5316 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[37 (1437917175356588032)]} 0 11
[junit4:junit4] 2> 2497730 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10037 (1437917175358685184)]} 0 10
[junit4:junit4] 2> 2497804 T5296 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[38 (1437917175444668416)]} 0 2
[junit4:junit4] 2> 2497810 T5297 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10038 (1437917175450959872)]} 0 2
[junit4:junit4] 2> 2497824 T5329 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[38 (1437917175457251328)]} 0 3
[junit4:junit4] 2> 2497827 T5314 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[38 (1437917175457251328)]} 0 14
[junit4:junit4] 2> 2497837 T5332 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10038 (1437917175460397056)]} 0 6
[junit4:junit4] 2> 2497843 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10038 (1437917175460397056)]} 0 26
[junit4:junit4] 2> 2497910 T5294 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[39 (1437917175557914624)]} 0 0
[junit4:junit4] 2> 2497914 T5292 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10039 (1437917175561060352)]} 0 1
[junit4:junit4] 2> 2497922 T5331 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10039 (1437917175566303232)]} 0 1
[junit4:junit4] 2> 2497923 T5331 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:53256/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[39 (1437917175565254656)]} 0 0
[junit4:junit4] 2> 2497925 T5312 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10039 (1437917175566303232)]} 0 7
[junit4:junit4] 2> 2497926 T5313 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[39 (1437917175565254656)]} 0 10
[junit4:junit4] 2> 2497943 T5302 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2497945 T5302 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:53259__collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:53259_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53259"}
[junit4:junit4] 2> 2497959 T5337 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2497960 T5321 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2497959 T5308 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2497959 T5301 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2498018 T5296 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10040 (1437917175670112256)]} 0 2
[junit4:junit4] 2> 2498018 T5295 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[40 (1437917175670112257)]} 0 2
[junit4:junit4] 2> 2498023 T5316 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10040 (1437917175675355136)]} 0 1
[junit4:junit4] 2> 2498024 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[40 (1437917175675355137)]} 0 2
[junit4:junit4] 2> 2498089 T5294 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[35 (-1437917175744561153)]} 0 1
[junit4:junit4] 2> 2498089 T5297 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10035 (-1437917175744561152)]} 0 1
[junit4:junit4] 2> 2498095 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[35 (-1437917175751901184)]} 0 0
[junit4:junit4] 2> 2498095 T5314 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10035 (-1437917175751901185)]} 0 1
[junit4:junit4] 2> 2498102 T5293 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[41 (1437917175757144064)]} 0 2
[junit4:junit4] 2> 2498103 T5292 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10041 (1437917175757144065)]} 0 3
[junit4:junit4] 2> 2498109 T5312 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[41 (1437917175764484096)]} 0 2
[junit4:junit4] 2> 2498112 T5312 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10041 (1437917175768678400)]} 0 1
[junit4:junit4] 2> 2498195 T5296 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[42 (1437917175854661632)]} 0 2
[junit4:junit4] 2> 2498198 T5296 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10042 (1437917175858855936)]} 0 1
[junit4:junit4] 2> 2498204 T5316 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[42 (1437917175865147392)]} 0 1
[junit4:junit4] 2> 2498207 T5316 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10042 (1437917175868293120)]} 0 1
[junit4:junit4] 2> 2498286 T5297 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10043 (1437917175950082049)]} 0 2
[junit4:junit4] 2> 2498286 T5294 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[43 (1437917175950082048)]} 0 2
[junit4:junit4] 2> 2498292 T5314 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10043 (1437917175956373504)]} 0 2
[junit4:junit4] 2> 2498293 T5313 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[43 (1437917175957422080)]} 0 2
[junit4:junit4] 2> 2498313 T5293 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10044 (1437917175978393600)]} 0 2
[junit4:junit4] 2> 2498317 T5293 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[44 (1437917175982587904)]} 0 2
[junit4:junit4] 2> 2498321 T5312 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10044 (1437917175987830784)]} 0 1
[junit4:junit4] 2> 2498324 T5316 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[44 (1437917175990976512)]} 0 1
[junit4:junit4] 2> 2498408 T5296 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10045 (1437917176079056896)]} 0 1
[junit4:junit4] 2> 2498410 T5297 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[45 (1437917176082202624)]} 0 0
[junit4:junit4] 2> 2498413 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10045 (1437917176085348352)]} 0 1
[junit4:junit4] 2> 2498417 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[45 (1437917176088494080)]} 0 1
[junit4:junit4] 2> 2498464 T5294 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10046 (1437917176137777152)]} 0 2
[junit4:junit4] 2> 2498478 T5292 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[46 (1437917176152457216)]} 0 1
[junit4:junit4] 2> 2498479 T5314 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10046 (1437917176153505792)]} 0 1
[junit4:junit4] 2> 2498483 T5313 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[46 (1437917176157700096)]} 0 2
[junit4:junit4] 2> 2498513 T5295 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10047 (1437917176190205952)]} 0 1
[junit4:junit4] 2> 2498518 T5312 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10047 (1437917176194400256)]} 0 1
[junit4:junit4] 2> 2498519 T5293 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[47 (1437917176194400256)]} 0 2
[junit4:junit4] 2> 2498524 T5316 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[47 (1437917176200691712)]} 0 1
[junit4:junit4] 2> 2498594 T5296 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10037 (-1437917176275140608)]} 0 0
[junit4:junit4] 2> 2498598 T5297 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[37 (-1437917176279334912)]} 0 0
[junit4:junit4] 2> 2498599 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10037 (-1437917176279334912)]} 0 1
[junit4:junit4] 2> 2498602 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[37 (-1437917176283529216)]} 0 0
[junit4:junit4] 2> 2498603 T5294 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10048 (1437917176283529216)]} 0 1
[junit4:junit4] 2> 2498607 T5292 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[48 (1437917176287723520)]} 0 1
[junit4:junit4] 2> 2498608 T5313 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10048 (1437917176289820672)]} 0 0
[junit4:junit4] 2> 2498613 T5312 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[48 (1437917176294014976)]} 0 1
[junit4:junit4] 2> 2498633 T5295 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10049 (1437917176314986496)]} 0 1
[junit4:junit4] 2> 2498637 T5293 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[49 (1437917176318132224)]} 0 2
[junit4:junit4] 2> 2498640 T5316 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10049 (1437917176320229376)]} 0 3
[junit4:junit4] 2> 2498651 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[49 (1437917176332812288)]} 0 2
[junit4:junit4] 2> 2498724 T5296 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10050 (1437917176411455488)]} 0 0
[junit4:junit4] 2> 2498728 T5294 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[50 (1437917176414601216)]} 0 1
[junit4:junit4] 2> 2498729 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10050 (1437917176415649792)]} 0 1
[junit4:junit4] 2> 2498732 T5314 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[50 (1437917176419844096)]} 0 1
[junit4:junit4] 2> 2498803 T5292 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10038 (-1437917176494292992)]} 0 0
[junit4:junit4] 2> 2498806 T5295 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[38 (-1437917176497438720)]} 0 1
[junit4:junit4] 2> 2498808 T5313 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10038 (-1437917176499535872)]} 0 0
[junit4:junit4] 2> 2498820 T5312 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[38 (-1437917176512118784)]} 0 0
[junit4:junit4] 2> 2498822 T5293 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10051 (1437917176513167360)]} 0 1
[junit4:junit4] 2> 2498825 T5297 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[51 (1437917176516313088)]} 0 1
[junit4:junit4] 2> 2498827 T5316 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10051 (1437917176518410240)]} 0 1
[junit4:junit4] 2> 2498829 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[51 (1437917176521555968)]} 0 0
[junit4:junit4] 2> 2498837 T5296 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10039 (-1437917176528896000)]} 0 1
[junit4:junit4] 2> 2498837 T5294 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[39 (-1437917176529944576)]} 0 0
[junit4:junit4] 2> 2498842 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10039 (-1437917176535187456)]} 0 0
[junit4:junit4] 2> 2498845 T5314 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[39 (-1437917176538333184)]} 0 1
[junit4:junit4] 2> 2498850 T5292 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10052 (1437917176541478912)]} 0 2
[junit4:junit4] 2> 2498853 T5292 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[52 (1437917176545673216)]} 0 2
[junit4:junit4] 2> 2498860 T5313 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10052 (1437917176550916096)]} 0 3
[junit4:junit4] 2> 2498861 T5312 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[52 (1437917176551964672)]} 0 3
[junit4:junit4] 2> 2498904 T5293 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10040 (-1437917176600199168)]} 0 1
[junit4:junit4] 2> 2498906 T5293 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[40 (-1437917176602296320)]} 0 0
[junit4:junit4] 2> 2498918 T5316 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10040 (-1437917176614879232)]} 0 4
[junit4:junit4] 2> 2498921 T5316 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[40 (-1437917176618024960)]} 0 0
[junit4:junit4] 2> 2498926 T5296 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10053 (1437917176621170688)]} 0 2
[junit4:junit4] 2> 2498927 T5294 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[53 (1437917176622219264)]} 0 2
[junit4:junit4] 2> 2498932 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10053 (1437917176628510720)]} 0 2
[junit4:junit4] 2> 2498933 T5314 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[53 (1437917176628510721)]} 0 2
[junit4:junit4] 2> C1214_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=down, core=collection1, collection=collection1, node_name=127.0.0.1:53259_, base_url=http://127.0.0.1:53259}
[junit4:junit4] 2> 2498981 T5340 C1214 P53259 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:53256/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 2498982 T5340 C1214 P53259 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:53259 START replicas=[http://127.0.0.1:53256/collection1/] nUpdates=100
[junit4:junit4] 2> 2498984 T5340 C1214 P53259 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 2498995 T5340 C1214 P53259 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 2498993 T5279 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 2498997 T5279 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 2498995 T5340 C1214 P53259 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 2498998 T5340 C1214 P53259 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 2498998 T5340 C1214 P53259 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1371304659326/jetty2/tlog/tlog.0000000000000000000 refcount=1}}
[junit4:junit4] 2> 2498999 T5340 C1214 P53259 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:53256/collection1/. core=collection1
[junit4:junit4] 2> 2499003 T5340 C1214 P53259 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 2498997 T5312 C1216 P53256 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 2499008 T5279 oasc.RecoveryStrategy.close WARN Stopping recovery for zkNodeName=127.0.0.1:53259__collection1core=collection1
[junit4:junit4] 2> 2499013 T5292 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[54 (1437917176713445376)]} 0 1
[junit4:junit4] 2> 2499025 T5295 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10054 (1437917176713445377)]} 0 13
[junit4:junit4] 2> 2499038 T5314 C1216 P53256 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 2499039 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10054 (1437917176740708352)]} 0 2
[junit4:junit4] 2> 2499044 T5316 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[54 (1437917176742805504)]} 0 7
[junit4:junit4] 2> 2499064 T5314 C1216 P53256 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@a167e0c lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bf4407c),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@a167e0c lockFactory=org.apache.lucene.store.NativeFSLockFactory@4bf4407c),segFN=segments_2,generation=2}
[junit4:junit4] 2> 2499068 T5314 C1216 P53256 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 2499091 T5314 C1216 P53256 oass.SolrIndexSearcher.<init> Opening Searcher@2a214064 realtime
[junit4:junit4] 2> 2499092 T5314 C1216 P53256 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 2499093 T5314 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 55
[junit4:junit4] 2> 2499095 T5340 C1214 P53259 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 2499096 T5340 C1214 P53259 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 2499101 T5312 C1216 P53256 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 2499102 T5312 C1216 P53256 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2
[junit4:junit4] 2> 2499103 T5340 C1214 P53259 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 2499103 T5340 C1214 P53259 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 2499104 T5340 C1214 P53259 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 2499107 T5315 C1216 P53256 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 2499108 T5340 C1214 P53259 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 5
[junit4:junit4] 2> 2499109 T5340 C1214 P53259 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371304659326/jetty2/index.20130615085800647
[junit4:junit4] 2> 2499109 T5340 C1214 P53259 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@3660fdc7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7510981d) fullCopy=false
[junit4:junit4] 2> 2499112 T5317 C1216 P53256 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=_0.cfs&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 2499117 T5316 C1216 P53256 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=_0.cfe&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 2499122 T5314 C1216 P53256 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1
[junit4:junit4] 2> 2499126 T5313 C1216 P53256 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=_0_1.del&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 2499132 T5293 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10055 (1437917176838225920)]} 0 1
[junit4:junit4] 2> 2499134 T5312 C1216 P53256 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=_0.si&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 2499140 T5296 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[55 (1437917176846614528)]} 0 1
[junit4:junit4] 2> 2499141 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10055 (1437917176844517376)]} 0 4
[junit4:junit4] 2> 2499141 T5340 C1214 P53259 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 2499144 T5340 C1214 P53259 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 2499145 T5340 C1214 P53259 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 2499145 T5340 C1214 P53259 oasu.DefaultSolrCoreState.newIndexWriter Closing old IndexWriter... core=collection1
[junit4:junit4] 2> 2499145 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[55 (1437917176851857408)]} 0 1
[junit4:junit4] 2> 2499164 T5340 C1214 P53259 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@68baee3f lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cb1c2c4),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@68baee3f lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cb1c2c4),segFN=segments_2,generation=2}
[junit4:junit4] 2> 2499165 T5340 C1214 P53259 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 2499170 T5340 C1214 P53259 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@68baee3f lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cb1c2c4),segFN=segments_2,generation=2}
[junit4:junit4] 2> 2499170 T5340 C1214 P53259 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 2499172 T5340 C1214 P53259 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 2499175 T5340 C1214 P53259 oass.SolrIndexSearcher.<init> Opening Searcher@5154c4a3 main
[junit4:junit4] 2> 2499176 T5339 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5154c4a3 main{StandardDirectoryReader(segments_2:3:nrt _0(4.4):c36/8)}
[junit4:junit4] 2> 2499177 T5340 C1214 P53259 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371304659326/jetty2/index.20130615085800647 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371304659326/jetty2/index.20130615085800647;done=true>>]
[junit4:junit4] 2> 2499177 T5340 C1214 P53259 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371304659326/jetty2/index.20130615085800647
[junit4:junit4] 2> 2499178 T5340 C1214 P53259 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.RecoveryZkTest-1371304659326/jetty2/index.20130615085800647
[junit4:junit4] 2> 2499182 T5340 C1214 P53259 oasc.RecoveryStrategy.replay Replaying buffered documents. core=collection1
[junit4:junit4] 2> 2499183 T5348 C1214 P53259 oasu.UpdateLog$LogReplayer.doReplay WARN Starting log replay tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1371304659326/jetty2/tlog/tlog.0000000000000000000 refcount=2} active=true starting pos=4041
[junit4:junit4] 2> 2499183 T5348 C1214 P53259 oasu.DirectUpdateHandler2.commit start commit{flags=2,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 2499187 T5348 C1214 P53259 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@68baee3f lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cb1c2c4),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@68baee3f lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cb1c2c4),segFN=segments_3,generation=3}
[junit4:junit4] 2> 2499187 T5348 C1214 P53259 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 2499188 T5348 C1214 P53259 oass.SolrIndexSearcher.<init> Opening Searcher@5302b72e main
[junit4:junit4] 2> 2499189 T5348 C1214 P53259 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 2499190 T5339 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5302b72e main{StandardDirectoryReader(segments_2:3:nrt _0(4.4):c36/8)}
[junit4:junit4] 2> 2499191 T5348 C1214 P53259 oasup.LogUpdateProcessor.finish [collection1] {} 0 8
[junit4:junit4] 2> 2499192 T5348 C1214 P53259 oasu.UpdateLog$LogReplayer.run WARN Log replay finished. recoveryInfo=RecoveryInfo{adds=0 deletes=0 deleteByQuery=0 errors=0 positionOfStart=4041}
[junit4:junit4] 2> 2499193 T5340 C1214 P53259 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 2499193 T5340 C1214 P53259 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 2499193 T5340 C1214 P53259 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 2499194 T5294 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10056 (1437917176903237632)]} 0 1
[junit4:junit4] 2> 2499198 T5292 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[56 (1437917176907431936)]} 0 1
[junit4:junit4] 2> 2499199 T5340 C1214 P53259 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 2499201 T5279 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@7cbce64a
[junit4:junit4] 2> 2499201 T5316 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10056 (1437917176910577664)]} 0 1
[junit4:junit4] 2> 2499211 T5316 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[56 (1437917176921063424)]} 0 1
[junit4:junit4] 2> 2499217 T5279 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=36,cumulative_deletesById=14,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 2499218 T5279 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 2499219 T5279 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 2499220 T5279 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 2499225 T5279 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 2499226 T5279 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 2499227 T5279 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371304659326/jetty2 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371304659326/jetty2;done=false>>]
[junit4:junit4] 2> 2499227 T5279 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371304659326/jetty2
[junit4:junit4] 2> 2499228 T5279 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371304659326/jetty2/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371304659326/jetty2/index;done=false>>]
[junit4:junit4] 2> 2499229 T5279 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371304659326/jetty2/index
[junit4:junit4] 2> 2499240 T5301 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2499240 T5321 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2499243 T5308 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 2499244 T5308 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2499249 T5301 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 2499250 T5321 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 2499251 T5295 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[41 (-1437917176964055040)]} 0 1
[junit4:junit4] 2> 2499251 T5293 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10041 (-1437917176964055041)]} 0 0
[junit4:junit4] 2> 2499255 T5313 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[41 (-1437917176968249344)]} 0 0
[junit4:junit4] 2> 2499259 T5312 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10041 (-1437917176969297920)]} 0 3
[junit4:junit4] 2> 2499259 T5279 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 2> 2499264 T5297 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[57 (1437917176976637952)]} 0 1
[junit4:junit4] 2> 2499265 T5296 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10057 (1437917176977686528)]} 0 1
[junit4:junit4] 2> 2499268 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[57 (1437917176981880832)]} 0 0
[junit4:junit4] 2> 2499269 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10057 (1437917176981880833)]} 0 1
[junit4:junit4] 2> 2499347 T5292 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[58 (1437917177063669760)]} 0 1
[junit4:junit4] 2> 2499348 T5294 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10058 (1437917177063669761)]} 0 2
[junit4:junit4] 2> 2499353 T5316 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[58 (1437917177069961216)]} 0 2
[junit4:junit4] 2> 2499356 T5314 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10058 (1437917177073106944)]} 0 2
[junit4:junit4] 2> 2499447 T5295 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[59 (1437917177167478784)]} 0 2
[junit4:junit4] 2> 2499450 T5293 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10059 (1437917177171673088)]} 0 1
[junit4:junit4] 2> 2499455 T5313 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[59 (1437917177174818816)]} 0 3
[junit4:junit4] 2> 2499463 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10059 (1437917177185304576)]} 0 1
[junit4:junit4] 2> 2499483 T5297 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[60 (1437917177206276096)]} 0 2
[junit4:junit4] 2> 2499485 T5302 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2499491 T5302 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:53259__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53259_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53259"}
[junit4:junit4] 2> 2499491 T5317 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[60 (1437917177214664704)]} 0 2
[junit4:junit4] 2> 2499493 T5296 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10060 (1437917177216761856)]} 0 1
[junit4:junit4] 2> 2499499 T5316 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10060 (1437917177222004736)]} 0 2
[junit4:junit4] 2> 2499505 T5302 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:53259__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53259_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53259"}
[junit4:junit4] 2> 2499518 T5279 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 2499519 T5308 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2499519 T5301 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2499521 T5321 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2499533 T5279 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53259
[junit4:junit4] 2> 2499536 T5279 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 2499537 T5279 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 2499537 T5279 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371304671870
[junit4:junit4] 2> 2499538 T5279 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371304671870/solr.xml
[junit4:junit4] 2> 2499539 T5279 oasc.CoreContainer.<init> New CoreContainer 1322205208
[junit4:junit4] 2> 2499540 T5279 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371304671870/'
[junit4:junit4] 2> 2499540 T5279 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371304671870/'
[junit4:junit4] 2> 2499569 T5292 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10061 (1437917177295405056)]} 0 2
[junit4:junit4] 2> 2499573 T5295 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[61 (1437917177300647936)]} 0 1
[junit4:junit4] 2> 2499577 T5314 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10061 (1437917177303793664)]} 0 2
[junit4:junit4] 2> 2499583 T5312 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[61 (1437917177311133696)]} 0 1
[junit4:junit4] 2> 2499666 T5293 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10043 (-1437917177399214080)]} 0 1
[junit4:junit4] 2> 2499671 T5297 C1215 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[43 (-1437917177404456960)]} 0 0
[junit4:junit4] 2> 2499671 T5313 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10043 (-1437917177404456960)]} 0 0
[junit4:junit4] 2> 2499675 T5315 C1216 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[43 (-1437917177408651264)]} 0 1
[junit4:junit4] 2> ASYNC NEW_CORE C1217 name=collection1 org.apache.solr.core.SolrCore@6eea934e url=http://127.0.0.1:53252/collection1 node=127.0.0.1:53252_ C1217_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:53252_, base_url=http://127.0.0.1:53252, leader=true}
[junit4:junit4] 2> 2499693 T5296 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10062 (1437917177426477056)]} 0 1
[junit4:junit4] 2> 2499703 T5296 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[62 (1437917177433817088)]} 0 5
[junit4:junit4] 2> ASYNC NEW_CORE C1218 name=collection1 org.apache.solr.core.SolrCore@26e891ef url=http://127.0.0.1:53256/collection1 node=127.0.0.1:53256_ C1218_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:53256_, base_url=http://127.0.0.1:53256, leader=true}
[junit4:junit4] 2> 2499715 T5317 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10062 (1437917177448497152)]} 0 3
[junit4:junit4] 2> 2499718 T5316 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[62 (1437917177452691456)]} 0 1
[junit4:junit4] 2> 2499767 T5295 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10044 (-1437917177505120256)]} 0 0
[junit4:junit4] 2> 2499767 T5292 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[44 (-1437917177505120257)]} 0 0
[junit4:junit4] 2> 2499782 T5314 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10044 (-1437917177511411712)]} 0 10
[junit4:junit4] 2> 2499784 T5312 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[44 (-1437917177522946048)]} 0 0
[junit4:junit4] 2> 2499790 T5293 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10063 (1437917177528188928)]} 0 1
[junit4:junit4] 2> 2499803 T5313 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10063 (1437917177538674688)]} 0 5
[junit4:junit4] 2> 2499812 T5297 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[63 (1437917177551257600)]} 0 2
[junit4:junit4] 2> 2499819 T5315 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[63 (1437917177557549056)]} 0 2
[junit4:junit4] 2> 2499843 T5294 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10064 (1437917177582714880)]} 0 2
[junit4:junit4] 2> 2499856 T5316 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10064 (1437917177595297792)]} 0 3
[junit4:junit4] 2> 2499872 T5296 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[64 (1437917177613123584)]} 0 2
[junit4:junit4] 2> 2499888 T5314 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[64 (1437917177630949376)]} 0 2
[junit4:junit4] 2> 2499972 T5295 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10065 (1437917177719029760)]} 0 2
[junit4:junit4] 2> 2499981 T5312 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10065 (1437917177725321216)]} 0 5
[junit4:junit4] 2> 2499991 T5279 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 2499992 T5279 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 2499992 T5279 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 2499993 T5279 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 2499994 T5279 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 2499999 T5293 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[65 (1437917177746292736)]} 0 2
[junit4:junit4] 2> 2500005 T5313 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[65 (1437917177752584192)]} 0 2
[junit4:junit4] 2> 2499998 T5279 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 2500009 T5279 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 2500009 T5279 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 2500010 T5279 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 2500011 T5279 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 2500035 T5297 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10066 (1437917177784041472)]} 0 2
[junit4:junit4] 2> 2500046 T5315 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10066 (1437917177795575808)]} 0 2
[junit4:junit4] 2> 2500058 T5294 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[66 (1437917177808158720)]} 0 2
[junit4:junit4] 2> 2500068 T5317 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[66 (1437917177815498752)]} 0 5
[junit4:junit4] 2> 2500086 T5279 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 2500087 T5279 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53249/solr
[junit4:junit4] 2> 2500088 T5279 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 2500092 T5279 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2500102 T5359 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@47df46a1 name:ZooKeeperConnection Watcher:127.0.0.1:53249 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2500104 T5279 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2500107 T5296 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10048 (-1437917177861636096)]} 0 0
[junit4:junit4] 2> 2500112 T5316 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10048 (-1437917177866878976)]} 0 0
[junit4:junit4] 2> 2500119 T5279 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 2500131 T5292 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10067 (1437917177885753344)]} 0 1
[junit4:junit4] 2> 2500149 T5295 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[48 (-1437917177905676288)]} 0 0
[junit4:junit4] 2> 2500149 T5314 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10067 (1437917177902530560)]} 0 4
[junit4:junit4] 2> 2500156 T5312 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[48 (-1437917177911967744)]} 0 1
[junit4:junit4] 2> 2500164 T5293 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[67 (1437917177919307776)]} 0 3
[junit4:junit4] 2> 2500171 T5313 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[67 (1437917177926647808)]} 0 2
[junit4:junit4] 2> 2500171 T5279 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2500180 T5361 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f01120d name:ZooKeeperConnection Watcher:127.0.0.1:53249/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2500181 T5279 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2500194 T5297 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10049 (-1437917177951813632)]} 0 1
[junit4:junit4] 2> 2500200 T5315 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10049 (-1437917177959153664)]} 0 1
[junit4:junit4] 2> 2500206 T5294 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10068 (1437917177964396544)]} 0 2
[junit4:junit4] 2> 2500209 T5279 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 2500212 T5317 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10068 (1437917177970688000)]} 0 1
[junit4:junit4] 2> 2500215 T5296 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[49 (-1437917177974882304)]} 0 0
[junit4:junit4] 2> 2500221 T5316 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[49 (-1437917177981173760)]} 0 1
[junit4:junit4] 2> 2500229 T5292 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[68 (1437917177987465216)]} 0 3
[junit4:junit4] 2> 2500237 T5314 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[68 (1437917177994805248)]} 0 3
[junit4:junit4] 2> 2500277 T5295 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10069 (1437917178038845440)]} 0 2
[junit4:junit4] 2> 2500284 T5312 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10069 (1437917178044088320)]} 0 3
[junit4:junit4] 2> 2500295 T5293 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[69 (1437917178057719808)]} 0 2
[junit4:junit4] 2> 2500299 T5315 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[69 (1437917178061914112)]} 0 1
[junit4:junit4] 2> 2500389 T5297 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10070 (1437917178155237376)]} 0 2
[junit4:junit4] 2> 2500393 T5317 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10070 (1437917178161528832)]} 0 1
[junit4:junit4] 2> 2500404 T5296 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[70 (1437917178172014592)]} 0 1
[junit4:junit4] 2> 2500408 T5316 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[70 (1437917178176208896)]} 0 1
[junit4:junit4] 2> 2500432 T5292 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10052 (-1437917178201374720)]} 0 1
[junit4:junit4] 2> 2500437 T5314 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10052 (-1437917178207666176)]} 0 0
[junit4:junit4] 2> 2500444 T5295 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10071 (1437917178212909056)]} 0 2
[junit4:junit4] 2> 2500448 T5293 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[52 (-1437917178218151936)]} 0 0
[junit4:junit4] 2> 2500453 T5313 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10071 (1437917178222346240)]} 0 2
[junit4:junit4] 2> 2500454 T5313 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[52 (-1437917178225491968)]} 0 0
[junit4:junit4] 2> 2500460 T5294 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[71 (1437917178230734848)]} 0 2
[junit4:junit4] 2> 2500467 T5315 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[71 (1437917178238074880)]} 0 2
[junit4:junit4] 2> 2500522 T5297 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10072 (1437917178295746560)]} 0 2
[junit4:junit4] 2> 2500528 T5317 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10072 (1437917178302038016)]} 0 1
[junit4:junit4] 2> 2500535 T5296 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[72 (1437917178308329472)]} 0 2
[junit4:junit4] 2> 2500541 T5337 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 2500542 T5337 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2500543 T5337 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 2500542 T5316 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[72 (1437917178315669504)]} 0 2
[junit4:junit4] 2> 2500591 T5292 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10073 (1437917178367049728)]} 0 2
[junit4:junit4] 2> 2500597 T5314 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10073 (1437917178373341184)]} 0 2
[junit4:junit4] 2> 2500608 T5295 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[73 (1437917178384875520)]} 0 2
[junit4:junit4] 2> 2500614 T5313 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[73 (1437917178392215552)]} 0 2
[junit4:junit4] 2> 2500655 T5293 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10057 (-1437917178436255744)]} 0 0
[junit4:junit4] 2> 2500659 T5312 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10057 (-1437917178440450048)]} 0 0
[junit4:junit4] 2> 2500663 T5294 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10074 (1437917178444644352)]} 0 1
[junit4:junit4] 2> 2500668 T5315 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10074 (1437917178448838656)]} 0 1
[junit4:junit4] 2> 2500672 T5297 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[57 (-1437917178454081536)]} 0 0
[junit4:junit4] 2> 2500677 T5317 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[57 (-1437917178458275840)]} 0 0
[junit4:junit4] 2> 2500682 T5296 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[74 (1437917178462470144)]} 0 2
[junit4:junit4] 2> 2500688 T5316 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[74 (1437917178468761600)]} 0 2
[junit4:junit4] 2> 2500705 T5292 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10075 (1437917178486587392)]} 0 2
[junit4:junit4] 2> 2500711 T5313 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10075 (1437917178492878848)]} 0 2
[junit4:junit4] 2> 2500727 T5295 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[75 (1437917178509656064)]} 0 2
[junit4:junit4] 2> 2500734 T5312 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[75 (1437917178516996096)]} 0 2
[junit4:junit4] 2> 2500784 T5293 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10060 (-1437917178571522048)]} 0 0
[junit4:junit4] 2> 2500789 T5315 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10060 (-1437917178576764928)]} 0 0
[junit4:junit4] 2> 2500797 T5294 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10076 (1437917178582007808)]} 0 3
[junit4:junit4] 2> 2500805 T5296 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[60 (-1437917178593542144)]} 0 0
[junit4:junit4] 2> 2500806 T5317 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10076 (1437917178593542144)]} 0 1
[junit4:junit4] 2> 2500813 T5314 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[60 (-1437917178600882176)]} 0 1
[junit4:junit4] 2> 2500818 T5292 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[76 (1437917178607173632)]} 0 1
[junit4:junit4] 2> 2500825 T5316 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[76 (1437917178613465088)]} 0 2
[junit4:junit4] 2> 2500908 T5295 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10061 (-1437917178701545472)]} 0 1
[junit4:junit4] 2> 2500911 T5313 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10061 (-1437917178704691200)]} 0 0
[junit4:junit4] 2> 2500916 T5293 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10077 (1437917178708885504)]} 0 1
[junit4:junit4] 2> 2500919 T5312 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10077 (1437917178713079808)]} 0 0
[junit4:junit4] 2> 2500928 T5297 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[61 (-1437917178722516992)]} 0 0
[junit4:junit4] 2> 2500935 T5315 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[61 (-1437917178728808448)]} 0 1
[junit4:junit4] 2> 2500940 T5294 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[77 (1437917178733002752)]} 0 1
[junit4:junit4] 2> 2500945 T5317 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[77 (1437917178739294208)]} 0 2
[junit4:junit4] 2> 2500966 T5296 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10062 (-1437917178762362880)]} 0 0
[junit4:junit4] 2> 2500972 T5314 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10062 (-1437917178767605760)]} 0 2
[junit4:junit4] 2> 2500980 T5292 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10078 (1437917178775994368)]} 0 1
[junit4:junit4] 2> 2500986 T5316 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10078 (1437917178782285824)]} 0 2
[junit4:junit4] 2> 2500990 T5295 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[62 (-1437917178787528704)]} 0 0
[junit4:junit4] 2> 2500995 T5313 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[62 (-1437917178792771584)]} 0 0
[junit4:junit4] 2> 2501000 T5293 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[78 (1437917178796965888)]} 0 1
[junit4:junit4] 2> 2501006 T5312 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[78 (1437917178802208768)]} 0 2
[junit4:junit4] 2> 2501042 T5297 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10079 (1437917178836811776)]} 0 6
[junit4:junit4] 2> 2501049 T5315 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10079 (1437917178848346112)]} 0 2
[junit4:junit4] 2> 2501056 T5294 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[79 (1437917178855686144)]} 0 1
[junit4:junit4] 2> 2501060 T5317 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[79 (1437917178859880448)]} 0 1
[junit4:junit4] 2> 2501071 T5296 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10080 (1437917178871414784)]} 0 2
[junit4:junit4] 2> 2501075 T5314 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10080 (1437917178875609088)]} 0 1
[junit4:junit4] 2> 2501079 T5302 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2501082 T5292 C1217 P53252 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[80 (1437917178882949120)]} 0 2
[junit4:junit4] 2> 2501085 T5302 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:53259__collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:53259_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53259"}
[junit4:junit4] 2> 2501088 T5316 C1218 P53256 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[80 (1437917178889240576)]} 0 1
[junit4:junit4] 2> 2501100 T5361 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2501100 T5321 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has
[...truncated too long message...]
d, but Solr cannot talk to ZK
[junit4:junit4] 2> 2599672 T5301 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 2600577 T5279 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 2600578 T5279 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 2600581 T5279 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@253793c2
[junit4:junit4] 2> 2600592 T5279 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=43,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=52,cumulative_deletesById=23,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 2600593 T5279 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 2600594 T5279 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 2600594 T5279 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 2600597 T5279 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 2600598 T5279 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 2600599 T5279 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371304659326/jetty2 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371304659326/jetty2;done=false>>]
[junit4:junit4] 2> 2600599 T5279 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371304659326/jetty2
[junit4:junit4] 2> 2600600 T5279 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371304659326/jetty2/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371304659326/jetty2/index;done=false>>]
[junit4:junit4] 2> 2600600 T5279 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371304659326/jetty2/index
[junit4:junit4] 2> 2600602 T5373 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89869822154440714-127.0.0.1:53259_-n_0000000003) am no longer a leader.
[junit4:junit4] 2> 2600624 T5279 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 2> 2600705 T5279 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 2600710 T5279 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:53249 53249
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=RecoveryZkTest -Dtests.method=testDistribSearch -Dtests.seed=97D5D962CEE15B4C -Dtests.slow=true -Dtests.locale=fr_LU -Dtests.timezone=America/Panama -Dtests.file.encoding=US-ASCII
[junit4:junit4] FAILURE 123s | RecoveryZkTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: expected:<198> but was:<52>
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([97D5D962CEE15B4C:1633577AB9BE3B70]:0)
[junit4:junit4] > at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:108)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2> 2600879 T5279 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 123117 T5278 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4] 2> 2601909 T5361 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 2601914 T5361 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 2601914 T5361 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {range_facet_l=MockVariableIntBlock(baseBlockSize=11), text=Lucene41(blocksize=128), _version_=PostingsFormat(name=MockSep), rnd_b=PostingsFormat(name=Asserting), intDefault=PostingsFormat(name=MockSep), timestamp=MockVariableIntBlock(baseBlockSize=11), id=PostingsFormat(name=Asserting), a_t=MockVariableIntBlock(baseBlockSize=11), range_facet_sl=PostingsFormat(name=Asserting), range_facet_si=MockVariableIntBlock(baseBlockSize=11), other_tl1=PostingsFormat(name=MockSep), multiDefault=MockVariableIntBlock(baseBlockSize=11), a_si=PostingsFormat(name=MockSep)}, docValues:{}, sim=DefaultSimilarity, locale=fr_LU, timezone=America/Panama
[junit4:junit4] 2> NOTE: Mac OS X 10.8.4 x86_64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=2,threads=1,free=65131728,total=388759552
[junit4:junit4] 2> NOTE: All tests run in this JVM: [SortByFunctionTest, XsltUpdateRequestHandlerTest, RegexBoostProcessorTest, ChaosMonkeySafeLeaderTest, DOMUtilTest, TestWordDelimiterFilterFactory, ShardSplitTest, AlternateDirectoryTest, TestReversedWildcardFilterFactory, DirectSolrSpellCheckerTest, SystemInfoHandlerTest, DistributedSpellCheckComponentTest, TestLFUCache, SolrIndexConfigTest, TestArbitraryIndexDir, TestFieldCollectionResource, PluginInfoTest, ShowFileRequestHandlerTest, TestXIncludeConfig, TestPostingsSolrHighlighter, TestReloadAndDeleteDocs, BasicDistributedZk2Test, TestSolrXml, TestFiltering, TestShardHandlerFactory, CSVRequestHandlerTest, TestUpdate, WordBreakSolrSpellCheckerTest, ShardRoutingTest, TestRecovery, TestSolrDeletionPolicy1, ZkCLITest, SuggesterTest, TestQueryTypes, URLClassifyProcessorTest, TestJmxMonitoredMap, OverseerTest, SyncSliceTest, TestElisionMultitermQuery, SolrRequestParserTest, IndexSchemaTest, DistributedTermsComponentTest, TestStressReorder, CachingDirectoryFactoryTest, JsonLoaderTest, FieldMutatingUpdateProcessorTest, TestSolrCoreProperties, TermVectorComponentDistributedTest, DirectSolrConnectionTest, IndexBasedSpellCheckerTest, TestPhraseSuggestions, TestReload, TestPluginEnable, TestFunctionQuery, BasicDistributedZkTest, TestLMDirichletSimilarityFactory, SampleTest, TestPartialUpdateDeduplication, TestOmitPositions, TestComponentsName, TestReplicationHandler, FileUtilsTest, QueryEqualityTest, TestCoreContainer, ChaosMonkeyShardSplitTest, CoreContainerCoreInitFailuresTest, XmlUpdateRequestHandlerTest, SolrCmdDistributorTest, ZkControllerTest, TestFastLRUCache, MultiTermTest, EchoParamsTest, TestSchemaResource, TestPseudoReturnFields, TestQueryUtils, TestBinaryResponseWriter, TestPropInject, TestRTGBase, TestBadConfig, TestFieldTypeResource, TestSolrIndexConfig, UniqFieldsUpdateProcessorFactoryTest, TestMaxScoreQueryParser, RAMDirectoryFactoryTest, PathHierarchyTokenizerFactoryTest, ShardRoutingCustomTest, TestSchemaNameResource, OpenCloseCoreStressTest, SpatialFilterTest, ZkSolrClientTest, TestIndexingPerformance, TestLRUCache, TestPerFieldSimilarity, SolrInfoMBeanTest, TestDefaultSimilarityFactory, FileBasedSpellCheckerTest, TestIndexSearcher, TestUniqueKeyFieldResource, TestBinaryField, ConvertedLegacyTest, DocumentAnalysisRequestHandlerTest, TermVectorComponentTest, TestAnalyzedSuggestions, TestCSVLoader, SolrIndexSplitterTest, TestSolrQueryParser, SOLR749Test, TestSolrDeletionPolicy2, HighlighterConfigTest, UpdateRequestProcessorFactoryTest, IndexReaderFactoryTest, TestNumberUtils, AliasIntegrationTest, PreAnalyzedFieldTest, TestHashPartitioner, TestValueSourceCache, TestLMJelinekMercerSimilarityFactory, RequiredFieldsTest, PreAnalyzedUpdateProcessorTest, TestCodecSupport, TestZkChroot, QueryElevationComponentTest, TestClassNameShortening, BasicZkTest, JSONWriterTest, TestIBSimilarityFactory, DocValuesMultiTest, TestSearchPerf, SchemaVersionSpecificBehaviorTest, TestSchemaVersionResource, TestSurroundQueryParser, TestManagedSchema, DirectUpdateHandlerTest, ChaosMonkeyNothingIsSafeTest, LeaderElectionIntegrationTest, TestDistributedSearch, TestCloudManagedSchema, TestWriterPerf, TestDFRSimilarityFactory, CircularListTest, TestPropInjectDefaults, TestSerializedLuceneMatchVersion, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest, RecoveryZkTest]
[junit4:junit4] Completed in 124.20s, 1 test, 1 failure <<< FAILURES!
[...truncated 453 lines...]
BUILD FAILED
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/build.xml:392: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/build.xml:372: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build.xml:181: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/common-build.xml:437: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:1246: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:890: There were test failures: 297 suites, 1246 tests, 1 failure, 18 ignored (6 assumptions)
Total time: 121 minutes 13 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure