You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/05/30 15:15:49 UTC

[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_55) - Build # 10427 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10427/
Java: 32bit/jdk1.7.0_55 -server -XX:+UseParallelGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ReplicationFactorTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([511A59F9A569CC43]:0)


REGRESSION:  org.apache.solr.cloud.ReplicationFactorTest.testDistribSearch

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([511A59F9A569CC43]:0)




Build Log:
[...truncated 11798 lines...]
   [junit4] Suite: org.apache.solr.cloud.ReplicationFactorTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/init-core-data-001
   [junit4]   2> 756406 T3082 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /w/s
   [junit4]   2> 756411 T3082 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 756411 T3082 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 756412 T3083 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 756512 T3082 oasc.ZkTestServer.run start zk server on port:57711
   [junit4]   2> 756513 T3082 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 756583 T3089 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a6031e name:ZooKeeperConnection Watcher:127.0.0.1:57711 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 756583 T3082 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 756584 T3082 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 756590 T3082 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 756591 T3091 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b48f7b name:ZooKeeperConnection Watcher:127.0.0.1:57711/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 756592 T3082 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 756592 T3082 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 756596 T3082 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 756597 T3082 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 756599 T3082 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 756606 T3082 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 756607 T3082 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 756613 T3082 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 756613 T3082 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 756615 T3082 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 756616 T3082 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 756618 T3082 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 756618 T3082 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 756622 T3082 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 756622 T3082 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 756624 T3082 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 756625 T3082 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 756628 T3082 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 756628 T3082 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 756631 T3082 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 756631 T3082 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 756633 T3082 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 756633 T3082 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 756635 T3082 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 756636 T3082 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 756637 T3082 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 756637 T3082 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 756878 T3082 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 756880 T3082 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:37150
   [junit4]   2> 756880 T3082 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 756881 T3082 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 756881 T3082 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-002
   [junit4]   2> 756881 T3082 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-002/'
   [junit4]   2> 756895 T3082 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-002/solr.xml
   [junit4]   2> 756931 T3082 oasc.CoreContainer.<init> New CoreContainer 2867872
   [junit4]   2> 756931 T3082 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-002/]
   [junit4]   2> 756932 T3082 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 756933 T3082 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 756933 T3082 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 756933 T3082 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 756934 T3082 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 756934 T3082 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 756934 T3082 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 756934 T3082 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 756935 T3082 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 756935 T3082 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 756936 T3082 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 756936 T3082 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 756936 T3082 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57711/solr
   [junit4]   2> 756937 T3082 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 756937 T3082 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 756940 T3102 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f6b228 name:ZooKeeperConnection Watcher:127.0.0.1:57711 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 756940 T3082 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 756943 T3082 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 756944 T3104 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@dba1b7 name:ZooKeeperConnection Watcher:127.0.0.1:57711/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 756945 T3082 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 756946 T3082 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 756950 T3082 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 756953 T3082 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 756956 T3082 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 756959 T3082 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 756963 T3082 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 756965 T3082 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57811_w%2Fs
   [junit4]   2> 756967 T3082 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57811_w%2Fs
   [junit4]   2> 756971 T3082 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 756978 T3082 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 756984 T3082 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:57811_w%2Fs
   [junit4]   2> 756984 T3082 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 756987 T3082 oasc.Overseer.start Overseer (id=91845329425006595-127.0.0.1:57811_w%2Fs-n_0000000000) starting
   [junit4]   2> 756994 T3082 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 757000 T3106 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 757000 T3082 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 757002 T3082 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 757004 T3082 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 757005 T3105 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 757008 T3107 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 757009 T3107 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 757010 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 757010 T3107 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 757012 T3105 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 757013 T3105 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57811/w/s",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57811_w%2Fs",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 757014 T3105 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 757014 T3105 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 757017 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 757017 T3104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 758011 T3107 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 758012 T3107 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 758013 T3107 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 758013 T3107 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 758014 T3107 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 758015 T3107 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-002/collection1/'
   [junit4]   2> 758016 T3107 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 758016 T3107 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 758017 T3107 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 758058 T3107 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 758091 T3107 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 758093 T3107 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 758099 T3107 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 758378 T3107 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 758383 T3107 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 758385 T3107 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 758399 T3107 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 758402 T3107 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 758404 T3107 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 758405 T3107 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 758405 T3107 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 758405 T3107 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 758406 T3107 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 758406 T3107 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 758407 T3107 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 758407 T3107 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 758408 T3107 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-001/control/data/
   [junit4]   2> 758408 T3107 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@139ecdf
   [junit4]   2> 758409 T3107 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-001/control/data
   [junit4]   2> 758409 T3107 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-001/control/data/index/
   [junit4]   2> 758410 T3107 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 758410 T3107 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-001/control/data/index
   [junit4]   2> 758411 T3107 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=38, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=0.5703125, noCFSRatio=1.0]
   [junit4]   2> 758412 T3107 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@10642b8 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@10a2ca6),segFN=segments_1,generation=1}
   [junit4]   2> 758412 T3107 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 758415 T3107 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 758416 T3107 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 758416 T3107 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 758416 T3107 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 758416 T3107 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 758417 T3107 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 758417 T3107 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 758417 T3107 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 758418 T3107 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 758418 T3107 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 758418 T3107 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 758419 T3107 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 758419 T3107 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 758419 T3107 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 758419 T3107 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 758420 T3107 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 758420 T3107 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 758422 T3107 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 758425 T3107 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 758425 T3107 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 758425 T3107 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=15.2119140625, floorSegmentMB=0.314453125, forceMergeDeletesPctAllowed=29.847777813798476, segmentsPerTier=47.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 758426 T3107 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@10642b8 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@10a2ca6),segFN=segments_1,generation=1}
   [junit4]   2> 758426 T3107 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 758426 T3107 oass.SolrIndexSearcher.<init> Opening Searcher@701950[collection1] main
   [junit4]   2> 758427 T3107 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 758427 T3107 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 758428 T3107 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 758428 T3107 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 758428 T3107 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 758428 T3107 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 758429 T3107 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 758429 T3107 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 758429 T3107 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 758429 T3107 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 758431 T3108 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@701950[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 758434 T3107 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 758435 T3111 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:57811/w/s collection:control_collection shard:shard1
   [junit4]   2> 758435 T3082 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 758435 T3082 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 758435 T3111 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 758438 T3082 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 758439 T3114 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@382715 name:ZooKeeperConnection Watcher:127.0.0.1:57711/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 758440 T3082 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 758441 T3082 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 758441 T3111 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 758442 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 758442 T3111 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 758442 T3111 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 758442 T3082 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 758443 T3105 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C513 name=collection1 org.apache.solr.core.SolrCore@193da52 url=http://127.0.0.1:57811/w/s/collection1 node=127.0.0.1:57811_w%2Fs C513_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:57811/w/s, core=collection1, node_name=127.0.0.1:57811_w%2Fs}
   [junit4]   2> 758443 T3111 C513 P57811 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:57811/w/s/collection1/
   [junit4]   2> 758443 T3111 C513 P57811 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 758444 T3111 C513 P57811 oasc.SyncStrategy.syncToMe http://127.0.0.1:57811/w/s/collection1/ has no replicas
   [junit4]   2> 758444 T3111 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:57811/w/s/collection1/ shard1
   [junit4]   2> 758444 T3111 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 758444 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 758453 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 758455 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 758557 T3104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 758557 T3114 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 758604 T3111 oasc.ZkController.register We are http://127.0.0.1:57811/w/s/collection1/ and leader is http://127.0.0.1:57811/w/s/collection1/
   [junit4]   2> 758604 T3111 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:57811/w/s
   [junit4]   2> 758605 T3111 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 758605 T3111 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 758605 T3111 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 758606 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 758606 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 758606 T3111 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 758606 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 758607 T3105 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 758608 T3105 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57811/w/s",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57811_w%2Fs",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 758610 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 758675 T3082 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 758676 T3082 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 758679 T3082 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52139
   [junit4]   2> 758679 T3082 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 758680 T3082 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 758680 T3082 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-003
   [junit4]   2> 758680 T3082 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-003/'
   [junit4]   2> 758691 T3082 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-003/solr.xml
   [junit4]   2> 758711 T3104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 758711 T3114 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 758735 T3082 oasc.CoreContainer.<init> New CoreContainer 7724478
   [junit4]   2> 758735 T3082 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-003/]
   [junit4]   2> 758736 T3082 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 758736 T3082 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 758737 T3082 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 758737 T3082 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 758737 T3082 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 758738 T3082 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 758738 T3082 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 758738 T3082 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 758738 T3082 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 758739 T3082 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 758739 T3082 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 758740 T3082 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 758740 T3082 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57711/solr
   [junit4]   2> 758740 T3082 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 758741 T3082 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 758743 T3125 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@310d30 name:ZooKeeperConnection Watcher:127.0.0.1:57711 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 758743 T3082 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 758746 T3082 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 758748 T3127 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1352c71 name:ZooKeeperConnection Watcher:127.0.0.1:57711/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 758748 T3082 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 758756 T3082 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 759758 T3082 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51545_w%2Fs
   [junit4]   2> 759759 T3082 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51545_w%2Fs
   [junit4]   2> 759768 T3128 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 759768 T3128 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 759769 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 759769 T3128 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 759770 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 759770 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 759771 T3105 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 759772 T3105 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51545/w/s",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51545_w%2Fs",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 759772 T3105 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 759772 T3105 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 759774 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 759876 T3114 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 759876 T3127 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 759876 T3104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 760770 T3128 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 760770 T3128 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 760771 T3128 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 760771 T3128 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 760772 T3128 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 760772 T3128 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-003/collection1/'
   [junit4]   2> 760773 T3128 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 760773 T3128 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 760773 T3128 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 760805 T3128 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 760839 T3128 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 760840 T3128 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 760845 T3128 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 761114 T3128 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 761119 T3128 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 761121 T3128 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 761131 T3128 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 761133 T3128 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 761136 T3128 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 761136 T3128 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 761137 T3128 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 761137 T3128 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 761138 T3128 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 761138 T3128 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 761138 T3128 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 761138 T3128 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 761138 T3128 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-001/jetty1/
   [junit4]   2> 761139 T3128 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@139ecdf
   [junit4]   2> 761139 T3128 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-001/jetty1
   [junit4]   2> 761140 T3128 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-001/jetty1/index/
   [junit4]   2> 761140 T3128 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 761140 T3128 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-001/jetty1/index
   [junit4]   2> 761140 T3128 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=38, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=0.5703125, noCFSRatio=1.0]
   [junit4]   2> 761141 T3128 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@118564e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7843f0),segFN=segments_1,generation=1}
   [junit4]   2> 761141 T3128 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 761145 T3128 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 761146 T3128 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 761146 T3128 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 761146 T3128 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 761146 T3128 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 761147 T3128 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 761147 T3128 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 761147 T3128 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 761147 T3128 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 761148 T3128 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 761148 T3128 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 761149 T3128 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 761149 T3128 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 761149 T3128 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 761150 T3128 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 761150 T3128 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 761151 T3128 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 761153 T3128 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 761155 T3128 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 761155 T3128 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 761156 T3128 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=15.2119140625, floorSegmentMB=0.314453125, forceMergeDeletesPctAllowed=29.847777813798476, segmentsPerTier=47.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 761156 T3128 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@118564e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7843f0),segFN=segments_1,generation=1}
   [junit4]   2> 761157 T3128 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 761157 T3128 oass.SolrIndexSearcher.<init> Opening Searcher@1846c78[collection1] main
   [junit4]   2> 761157 T3128 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 761158 T3128 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 761158 T3128 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 761158 T3128 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 761159 T3128 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 761159 T3128 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 761159 T3128 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 761159 T3128 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 761159 T3128 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 761160 T3128 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 761162 T3129 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1846c78[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 761164 T3128 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 761165 T3132 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51545/w/s collection:collection1 shard:shard3
   [junit4]   2> 761165 T3082 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 761165 T3132 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 761165 T3082 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 761169 T3132 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 761170 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 761171 T3132 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 761171 T3132 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 761171 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 761171 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> ASYNC  NEW_CORE C514 name=collection1 org.apache.solr.core.SolrCore@f6c6db url=http://127.0.0.1:51545/w/s/collection1 node=127.0.0.1:51545_w%2Fs C514_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:51545/w/s, core=collection1, node_name=127.0.0.1:51545_w%2Fs}
   [junit4]   2> 761171 T3132 C514 P51545 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:51545/w/s/collection1/
   [junit4]   2> 761171 T3105 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 761171 T3132 C514 P51545 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 761172 T3132 C514 P51545 oasc.SyncStrategy.syncToMe http://127.0.0.1:51545/w/s/collection1/ has no replicas
   [junit4]   2> 761172 T3132 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:51545/w/s/collection1/ shard3
   [junit4]   2> 761172 T3132 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 761173 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 761175 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 761179 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 761280 T3104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 761280 T3127 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 761280 T3114 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 761326 T3132 oasc.ZkController.register We are http://127.0.0.1:51545/w/s/collection1/ and leader is http://127.0.0.1:51545/w/s/collection1/
   [junit4]   2> 761326 T3132 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51545/w/s
   [junit4]   2> 761326 T3132 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 761326 T3132 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 761327 T3132 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 761328 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 761328 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 761328 T3132 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 761328 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 761329 T3105 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 761329 T3105 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51545/w/s",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51545_w%2Fs",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 761331 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 761388 T3082 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 761389 T3082 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 761391 T3082 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44300
   [junit4]   2> 761391 T3082 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 761392 T3082 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 761392 T3082 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-004
   [junit4]   2> 761392 T3082 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-004/'
   [junit4]   2> 761403 T3082 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-004/solr.xml
   [junit4]   2> 761432 T3114 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 761433 T3104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 761433 T3127 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 761437 T3082 oasc.CoreContainer.<init> New CoreContainer 27163750
   [junit4]   2> 761437 T3082 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-004/]
   [junit4]   2> 761438 T3082 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 761438 T3082 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 761439 T3082 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 761439 T3082 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 761439 T3082 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 761439 T3082 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 761440 T3082 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 761440 T3082 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 761440 T3082 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 761441 T3082 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 761441 T3082 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 761441 T3082 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 761442 T3082 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57711/solr
   [junit4]   2> 761442 T3082 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 761443 T3082 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 761444 T3144 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f9324c name:ZooKeeperConnection Watcher:127.0.0.1:57711 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 761445 T3082 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 761447 T3082 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 761448 T3146 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@25a255 name:ZooKeeperConnection Watcher:127.0.0.1:57711/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 761448 T3082 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 761456 T3082 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 762458 T3082 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40648_w%2Fs
   [junit4]   2> 762467 T3082 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40648_w%2Fs
   [junit4]   2> 762482 T3147 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 762482 T3147 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 762484 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 762484 T3147 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 762484 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 762485 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 762487 T3105 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 762488 T3105 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:40648/w/s",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:40648_w%2Fs",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 762489 T3105 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 762489 T3105 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 762492 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 762595 T3114 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 762595 T3127 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 762595 T3146 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 762595 T3104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 763484 T3147 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 763485 T3147 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 763485 T3147 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 763485 T3147 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 763487 T3147 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 763487 T3147 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-004/collection1/'
   [junit4]   2> 763488 T3147 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 763488 T3147 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 763488 T3147 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 763523 T3147 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 763574 T3147 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 763575 T3147 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 763581 T3147 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 763862 T3147 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 763867 T3147 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 763869 T3147 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 763881 T3147 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 763884 T3147 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 763887 T3147 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 763888 T3147 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 763888 T3147 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 763888 T3147 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 763889 T3147 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 763889 T3147 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 763890 T3147 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 763890 T3147 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 763890 T3147 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-004/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-001/jetty2/
   [junit4]   2> 763890 T3147 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@139ecdf
   [junit4]   2> 763891 T3147 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-001/jetty2
   [junit4]   2> 763891 T3147 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-001/jetty2/index/
   [junit4]   2> 763891 T3147 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 763891 T3147 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-001/jetty2/index
   [junit4]   2> 763892 T3147 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=38, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=0.5703125, noCFSRatio=1.0]
   [junit4]   2> 763892 T3147 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@d70b6d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@b71128),segFN=segments_1,generation=1}
   [junit4]   2> 763892 T3147 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 763897 T3147 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 763898 T3147 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 763898 T3147 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 763898 T3147 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 763899 T3147 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 763899 T3147 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 763900 T3147 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 763900 T3147 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 763901 T3147 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 763901 T3147 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 763902 T3147 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 763902 T3147 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 763903 T3147 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 763903 T3147 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 763904 T3147 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 763904 T3147 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 763905 T3147 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 763909 T3147 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 763913 T3147 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 763913 T3147 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 763914 T3147 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=15.2119140625, floorSegmentMB=0.314453125, forceMergeDeletesPctAllowed=29.847777813798476, segmentsPerTier=47.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 763914 T3147 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@d70b6d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@b71128),segFN=segments_1,generation=1}
   [junit4]   2> 763914 T3147 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 763914 T3147 oass.SolrIndexSearcher.<init> Opening Searcher@480d2b[collection1] main
   [junit4]   2> 763915 T3147 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 763916 T3147 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 763916 T3147 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 763916 T3147 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 763917 T3147 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 763917 T3147 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 763917 T3147 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 763917 T3147 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 763918 T3147 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 763918 T3147 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 763920 T3148 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@480d2b[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 763922 T3147 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 763923 T3151 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40648/w/s collection:collection1 shard:shard1
   [junit4]   2> 763924 T3082 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 763924 T3151 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 763924 T3082 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 763928 T3151 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 763929 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 763929 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 763929 T3151 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 763930 T3151 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 763929 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 763930 T3105 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C515 name=collection1 org.apache.solr.core.SolrCore@aba5c6 url=http://127.0.0.1:40648/w/s/collection1 node=127.0.0.1:40648_w%2Fs C515_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:40648/w/s, core=collection1, node_name=127.0.0.1:40648_w%2Fs}
   [junit4]   2> 763930 T3151 C515 P40648 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:40648/w/s/collection1/
   [junit4]   2> 763930 T3151 C515 P40648 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 763931 T3151 C515 P40648 oasc.SyncStrategy.syncToMe http://127.0.0.1:40648/w/s/collection1/ has no replicas
   [junit4]   2> 763931 T3151 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:40648/w/s/collection1/ shard1
   [junit4]   2> 763931 T3151 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 763932 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 763941 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 763944 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 764046 T3114 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 764046 T3104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 764046 T3146 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 764046 T3127 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 764092 T3151 oasc.ZkController.register We are http://127.0.0.1:40648/w/s/collection1/ and leader is http://127.0.0.1:40648/w/s/collection1/
   [junit4]   2> 764092 T3151 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40648/w/s
   [junit4]   2> 764092 T3151 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 764092 T3151 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 764092 T3151 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 764094 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 764094 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 764094 T3151 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 764094 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 764095 T3105 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 764095 T3105 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:40648/w/s",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:40648_w%2Fs",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 764097 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 764152 T3082 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 764153 T3082 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 764155 T3082 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60612
   [junit4]   2> 764156 T3082 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 764156 T3082 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 764156 T3082 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-005
   [junit4]   2> 764157 T3082 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-005/'
   [junit4]   2> 764168 T3082 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-005/solr.xml
   [junit4]   2> 764199 T3114 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 764199 T3146 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 764199 T3127 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 764199 T3104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 764202 T3082 oasc.CoreContainer.<init> New CoreContainer 5535998
   [junit4]   2> 764202 T3082 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-005/]
   [junit4]   2> 764203 T3082 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 764203 T3082 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 764204 T3082 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 764204 T3082 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 764204 T3082 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 764204 T3082 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 764205 T3082 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 764205 T3082 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 764205 T3082 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 764206 T3082 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 764206 T3082 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 764206 T3082 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 764207 T3082 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57711/solr
   [junit4]   2> 764207 T3082 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 764208 T3082 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 764209 T3163 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d39f27 name:ZooKeeperConnection Watcher:127.0.0.1:57711 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 764210 T3082 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 764212 T3082 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 764213 T3165 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@193ab2e name:ZooKeeperConnection Watcher:127.0.0.1:57711/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 764213 T3082 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 764220 T3082 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 765224 T3082 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58297_w%2Fs
   [junit4]   2> 765226 T3082 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58297_w%2Fs
   [junit4]   2> 765240 T3166 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 765241 T3166 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 765242 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 765243 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 765242 T3166 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 765243 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 765245 T3105 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 765246 T3105 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:58297/w/s",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:58297_w%2Fs",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 765246 T3105 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 765246 T3105 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 765249 T3104 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 765352 T3127 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 765352 T3114 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 765352 T3146 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 765352 T3104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 765352 T3165 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 766243 T3166 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 766244 T3166 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 766244 T3166 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 766244 T3166 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 766245 T3166 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 766246 T3166 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-005/collection1/'
   [junit4]   2> 766246 T3166 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-005/collection1/lib/.svn/' to classloader
   [junit4]   2> 766246 T3166 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-005/collection1/lib/classes/' to classloader
   [junit4]   2> 766247 T3166 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ReplicationFactorTest-511A59F9A569CC43-001/tempDir-005/collection1/lib/README' to classloader
   [junit4]   2> 766278 T3166 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 766313 T3166 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 766314 T3166 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 766319 T3166 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 766592 T3166 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, re

[...truncated too long message...]

Socket.<init>(Socket.java:208)
   [junit4]   2> 	        at org.apache.solr.cloud.ZkTestServer.send4LetterWord(ZkTestServer.java:314)
   [junit4]   2> 	        at org.apache.solr.cloud.ZkTestServer.waitForServerDown(ZkTestServer.java:275)
   [junit4]   2> 	        at org.apache.solr.cloud.ZkTestServer$ZKServerMain.shutdown(ZkTestServer.java:126)
   [junit4]   2> 	        at org.apache.solr.cloud.ZkTestServer.shutdown(ZkTestServer.java:262)
   [junit4]   2> 	        at org.apache.solr.cloud.AbstractDistribZkTestBase.tearDown(AbstractDistribZkTestBase.java:232)
   [junit4]   2> 	        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.tearDown(AbstractFullDistribZkTestBase.java:1427)
   [junit4]   2> 	        at org.apache.solr.cloud.ReplicationFactorTest.tearDown(ReplicationFactorTest.java:86)
   [junit4]   2> 	        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   2> 	        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	        at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:885)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   2> 	        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 	        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	   2) Thread[id=3164, name=TEST-ReplicationFactorTest.testDistribSearch-seed#[511A59F9A569CC43]-SendThread(127.0.0.1:57711), state=TIMED_WAITING, group=TGRP-ReplicationFactorTest]
   [junit4]   2> 	        at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:230)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1185)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1122)
   [junit4]   2> 7994083 T3082 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Lucene42, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=in, timezone=America/Chihuahua
   [junit4]   2> NOTE: Linux 3.8.0-41-generic i386/Oracle Corporation 1.7.0_55 (32-bit)/cpus=8,threads=3,free=144297616,total=306970624
   [junit4]   2> NOTE: All tests run in this JVM: [ExternalFileFieldSortTest, SliceStateTest, ScriptEngineTest, TestFunctionQuery, TestSchemaVersionResource, BadComponentTest, DeleteShardTest, TestRandomDVFaceting, BlockDirectoryTest, TestJmxMonitoredMap, TestLuceneMatchVersion, TestRemoteStreaming, UnloadDistributedZkTest, SuggesterFSTTest, DistributedExpandComponentTest, OverseerTest, MergeStrategyTest, TestUniqueKeyFieldResource, TestRequestStatusCollectionAPI, TestReplicationHandler, TestCursorMarkWithoutUniqueKey, DistributedQueryComponentCustomSortTest, TestCoreContainer, ZkControllerTest, TestFieldSortValues, InfoHandlerTest, TestSchemaNameResource, DistributedTermsComponentTest, TestBM25SimilarityFactory, HdfsBasicDistributedZk2Test, TestExceedMaxTermLength, ClusterStateTest, TestExtendedDismaxParser, TestFreeTextSuggestions, TestNonNRTOpen, TestOrdValues, PreAnalyzedUpdateProcessorTest, TestRealTimeGet, MBeansHandlerTest, HdfsChaosMonkeySafeLeaderTest, TestFieldResource, TestReversedWildcardFilterFactory, CircularListTest, TestPseudoReturnFields, SpatialFilterTest, WordBreakSolrSpellCheckerTest, JsonLoaderTest, TestSurroundQueryParser, TestLFUCache, FileBasedSpellCheckerTest, BasicDistributedZkTest, SyncSliceTest, SchemaVersionSpecificBehaviorTest, TestFaceting, TestPostingsSolrHighlighter, TestUtils, TestCharFilters, ResponseLogComponentTest, SpellPossibilityIteratorTest, UpdateParamsTest, TestSolrDeletionPolicy1, TestCollationFieldDocValues, ShowFileRequestHandlerTest, QueryParsingTest, TestFastLRUCache, TestLMJelinekMercerSimilarityFactory, SearchHandlerTest, ReturnFieldsTest, ReplicationFactorTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ReplicationFactorTest -Dtests.seed=511A59F9A569CC43 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=in -Dtests.timezone=America/Chihuahua -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J0 | ReplicationFactorTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([511A59F9A569CC43]:0)
   [junit4] Completed on J0 in 7237.73s, 1 test, 2 errors <<< FAILURES!

[...truncated 97 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1296: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:920: There were test failures: 406 suites, 1688 tests, 1 suite-level error, 1 error, 148 ignored (15 assumptions)

Total time: 172 minutes 15 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_55 -server -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_60-ea-b15) - Build # 10428 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10428/
Java: 64bit/jdk1.7.0_60-ea-b15 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

3 tests failed.
REGRESSION:  org.apache.solr.cloud.BasicDistributedZkTest.testDistribSearch

Error Message:
commitWithin did not work expected:<68> but was:<67>

Stack Trace:
java.lang.AssertionError: commitWithin did not work expected:<68> but was:<67>
	at __randomizedtesting.SeedInfo.seed([BD05FD5B9D125215:3CE37343EA4D3229]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.cloud.BasicDistributedZkTest.doTest(BasicDistributedZkTest.java:347)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:863)
	at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:745)


REGRESSION:  org.apache.solr.cloud.HttpPartitionTest.testDistribSearch

Error Message:
collection already exists: c8n_1x3

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: collection already exists: c8n_1x3
	at __randomizedtesting.SeedInfo.seed([BD05FD5B9D125215:3CE37343EA4D3229]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:554)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1500)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1460)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1514)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1456)
	at org.apache.solr.cloud.HttpPartitionTest.testRf3(HttpPartitionTest.java:223)
	at org.apache.solr.cloud.HttpPartitionTest.doTest(HttpPartitionTest.java:145)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:863)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:745)


REGRESSION:  org.apache.solr.cloud.ZkSolrClientTest.testReconnect

Error Message:
KeeperErrorCode = NodeExists for /collections/collection3

Stack Trace:
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection3
	at __randomizedtesting.SeedInfo.seed([BD05FD5B9D125215:64267429DA8B9F6A]:0)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:329)
	at org.apache.solr.cloud.ZkSolrClientTest.testReconnect(ZkSolrClientTest.java:136)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10718 lines...]
   [junit4] Suite: org.apache.solr.cloud.HttpPartitionTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/init-core-data-001
   [junit4]   2> 280206 T304 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /zp_tvz/hq
   [junit4]   2> 280216 T304 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 280217 T304 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 280218 T305 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 280317 T304 oasc.ZkTestServer.run start zk server on port:54428
   [junit4]   2> 280319 T304 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 288897 T308 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 8577ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
   [junit4]   2> 288900 T311 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@158ee1f1 name:ZooKeeperConnection Watcher:127.0.0.1:54428 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 288900 T304 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 288901 T304 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 288941 T304 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 288943 T313 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e919bec name:ZooKeeperConnection Watcher:127.0.0.1:54428/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 288944 T304 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 288945 T304 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 288950 T304 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 288954 T304 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 288957 T304 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 288960 T304 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 288961 T304 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 289047 T304 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 289048 T304 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 289052 T304 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 289052 T304 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 289055 T304 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 289055 T304 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 289058 T304 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 289059 T304 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 289061 T304 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 289061 T304 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 289063 T304 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 289064 T304 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 289066 T304 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 289067 T304 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 289069 T304 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 289069 T304 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 289072 T304 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 289072 T304 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 289074 T304 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 289075 T304 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 289347 T304 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 289361 T304 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40134
   [junit4]   2> 289362 T304 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 289362 T304 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 289363 T304 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-002
   [junit4]   2> 289363 T304 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-002/'
   [junit4]   2> 289383 T304 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-002/solr.xml
   [junit4]   2> 289420 T304 oasc.CoreContainer.<init> New CoreContainer 1148617275
   [junit4]   2> 289420 T304 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-002/]
   [junit4]   2> 289422 T304 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 289422 T304 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 289423 T304 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 289423 T304 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 289423 T304 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 289424 T304 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 289424 T304 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 289425 T304 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 289425 T304 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 289426 T304 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 289426 T304 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 289427 T304 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 289427 T304 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54428/solr
   [junit4]   2> 289427 T304 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 289428 T304 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 289452 T326 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4060129a name:ZooKeeperConnection Watcher:127.0.0.1:54428 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 289452 T304 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 289493 T304 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 289542 T328 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@31495ec1 name:ZooKeeperConnection Watcher:127.0.0.1:54428/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 289543 T304 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 289545 T304 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 289559 T304 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 289610 T304 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 289638 T304 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 289643 T304 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 289649 T304 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 289651 T304 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48768_zp_tvz%2Fhq
   [junit4]   2> 289653 T304 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48768_zp_tvz%2Fhq
   [junit4]   2> 289656 T304 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 289659 T304 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 289666 T304 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:48768_zp_tvz%2Fhq
   [junit4]   2> 289667 T304 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 289670 T304 oasc.Overseer.start Overseer (id=91846406863126531-127.0.0.1:48768_zp_tvz%2Fhq-n_0000000000) starting
   [junit4]   2> 289675 T304 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 289690 T330 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 289690 T304 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 289695 T304 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 289698 T304 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 289703 T329 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 289709 T331 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 289710 T331 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 289712 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 289712 T331 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 289715 T329 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 289716 T329 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48768/zp_tvz/hq",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:48768_zp_tvz%2Fhq",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 289716 T329 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 289717 T329 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 289720 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 289720 T328 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 290713 T331 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 290713 T331 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 290715 T331 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 290715 T331 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 290717 T331 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 290717 T331 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-002/collection1/'
   [junit4]   2> 290718 T331 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 290719 T331 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 290720 T331 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 290772 T331 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 290806 T331 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 290808 T331 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 290815 T331 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 291025 T331 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 291032 T331 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 291034 T331 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 291049 T331 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 291052 T331 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 291054 T331 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 291056 T331 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 291057 T331 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 291057 T331 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 291058 T331 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 291058 T331 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 291058 T331 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 291059 T331 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 291059 T331 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-001/control/data/
   [junit4]   2> 291059 T331 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e4bfd69
   [junit4]   2> 291060 T331 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-001/control/data
   [junit4]   2> 291060 T331 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-001/control/data/index/
   [junit4]   2> 291060 T331 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 291063 T331 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-001/control/data/index
   [junit4]   2> 291063 T331 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=32, maxMergeAtOnceExplicit=41, maxMergedSegmentMB=31.873046875, floorSegmentMB=0.3115234375, forceMergeDeletesPctAllowed=5.064936651571199, segmentsPerTier=43.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3054563728191859
   [junit4]   2> 291064 T331 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RateLimitedDirectoryWrapper(RAMDirectory@7f114a62 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7976c9c4)),segFN=segments_1,generation=1}
   [junit4]   2> 291064 T331 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 291068 T331 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 291068 T331 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 291068 T331 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 291068 T331 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 291069 T331 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 291069 T331 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 291069 T331 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 291070 T331 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 291070 T331 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 291070 T331 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 291071 T331 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 291071 T331 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 291071 T331 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 291072 T331 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 291072 T331 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 291072 T331 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 291073 T331 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 291075 T331 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 291078 T331 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 291078 T331 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 291079 T331 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=21, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7367731281060768]
   [junit4]   2> 291079 T331 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RateLimitedDirectoryWrapper(RAMDirectory@7f114a62 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7976c9c4)),segFN=segments_1,generation=1}
   [junit4]   2> 291080 T331 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 291080 T331 oass.SolrIndexSearcher.<init> Opening Searcher@465652da[collection1] main
   [junit4]   2> 291080 T331 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 291082 T331 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 291082 T331 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 291082 T331 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 291083 T331 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 291083 T331 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 291084 T331 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 291084 T331 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 291084 T331 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 291085 T331 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 291087 T332 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@465652da[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 291089 T331 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 291090 T335 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:48768/zp_tvz/hq collection:control_collection shard:shard1
   [junit4]   2> 291090 T304 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 291090 T304 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 291093 T304 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 291094 T335 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 291096 T338 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7845eeb name:ZooKeeperConnection Watcher:127.0.0.1:54428/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 291097 T304 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 291098 T304 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 291101 T304 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 291101 T335 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 291103 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 291103 T335 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 291104 T335 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C405 name=collection1 org.apache.solr.core.SolrCore@6cc70897 url=http://127.0.0.1:48768/zp_tvz/hq/collection1 node=127.0.0.1:48768_zp_tvz%2Fhq C405_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:48768/zp_tvz/hq, core=collection1, node_name=127.0.0.1:48768_zp_tvz%2Fhq}
   [junit4]   2> 291104 T335 C405 P48768 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:48768/zp_tvz/hq/collection1/
   [junit4]   2> 291104 T335 C405 P48768 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 291104 T329 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 291104 T335 C405 P48768 oasc.SyncStrategy.syncToMe http://127.0.0.1:48768/zp_tvz/hq/collection1/ has no replicas
   [junit4]   2> 291105 T335 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:48768/zp_tvz/hq/collection1/ shard1
   [junit4]   2> 291105 T335 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 291107 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 291115 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 291117 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 291219 T328 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 291220 T338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 291266 T335 oasc.ZkController.register We are http://127.0.0.1:48768/zp_tvz/hq/collection1/ and leader is http://127.0.0.1:48768/zp_tvz/hq/collection1/
   [junit4]   2> 291266 T335 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:48768/zp_tvz/hq
   [junit4]   2> 291266 T335 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 291266 T335 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 291267 T335 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 291269 T335 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 291269 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 291270 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 291270 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 291271 T329 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 291271 T329 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48768/zp_tvz/hq",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:48768_zp_tvz%2Fhq",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 291273 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 291375 T328 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 291375 T338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 291381 T304 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 291382 T304 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 291384 T304 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60479
   [junit4]   2> 291385 T304 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 291386 T304 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 291386 T304 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-003
   [junit4]   2> 291387 T304 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-003/'
   [junit4]   2> 291401 T304 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-003/solr.xml
   [junit4]   2> 291453 T304 oasc.CoreContainer.<init> New CoreContainer 338505595
   [junit4]   2> 291454 T304 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-003/]
   [junit4]   2> 291455 T304 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 291455 T304 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 291455 T304 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 291455 T304 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 291456 T304 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 291456 T304 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 291456 T304 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 291457 T304 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 291457 T304 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 291457 T304 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 291458 T304 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 291458 T304 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 291458 T304 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54428/solr
   [junit4]   2> 291459 T304 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 291459 T304 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 291461 T349 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@22883e96 name:ZooKeeperConnection Watcher:127.0.0.1:54428 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 291462 T304 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 291469 T304 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 291470 T351 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@76541506 name:ZooKeeperConnection Watcher:127.0.0.1:54428/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 291471 T304 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 291477 T304 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 292479 T304 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38340_zp_tvz%2Fhq
   [junit4]   2> 292481 T304 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38340_zp_tvz%2Fhq
   [junit4]   2> 292651 T352 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 292651 T352 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 292653 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 292654 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 292653 T352 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 292654 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 292655 T329 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 292656 T329 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:38340/zp_tvz/hq",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38340_zp_tvz%2Fhq",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 292656 T329 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 292656 T329 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 292658 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 292761 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 292761 T338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 292761 T328 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 293654 T352 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 293654 T352 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 293655 T352 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 293655 T352 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 293656 T352 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 293657 T352 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-003/collection1/'
   [junit4]   2> 293657 T352 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 293657 T352 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 293658 T352 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 293693 T352 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 293747 T352 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 293748 T352 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 293759 T352 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 294002 T352 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 294014 T352 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 294017 T352 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 294035 T352 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 294040 T352 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 294044 T352 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 294045 T352 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 294045 T352 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 294046 T352 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 294047 T352 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 294047 T352 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 294048 T352 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 294048 T352 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 294048 T352 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-001/jetty1/
   [junit4]   2> 294049 T352 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e4bfd69
   [junit4]   2> 294049 T352 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-001/jetty1
   [junit4]   2> 294050 T352 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-001/jetty1/index/
   [junit4]   2> 294050 T352 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 294051 T352 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-001/jetty1/index
   [junit4]   2> 294051 T352 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=32, maxMergeAtOnceExplicit=41, maxMergedSegmentMB=31.873046875, floorSegmentMB=0.3115234375, forceMergeDeletesPctAllowed=5.064936651571199, segmentsPerTier=43.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3054563728191859
   [junit4]   2> 294052 T352 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RateLimitedDirectoryWrapper(RAMDirectory@1a9009bb lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7361089b)),segFN=segments_1,generation=1}
   [junit4]   2> 294052 T352 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 294056 T352 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 294057 T352 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 294057 T352 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 294057 T352 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 294057 T352 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 294058 T352 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 294058 T352 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 294058 T352 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 294058 T352 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 294059 T352 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 294059 T352 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 294060 T352 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 294060 T352 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 294060 T352 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 294061 T352 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 294061 T352 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 294062 T352 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 294064 T352 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 294066 T352 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 294066 T352 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 294066 T352 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=21, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7367731281060768]
   [junit4]   2> 294067 T352 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RateLimitedDirectoryWrapper(RAMDirectory@1a9009bb lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7361089b)),segFN=segments_1,generation=1}
   [junit4]   2> 294067 T352 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 294068 T352 oass.SolrIndexSearcher.<init> Opening Searcher@65e92309[collection1] main
   [junit4]   2> 294068 T352 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 294069 T352 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 294069 T352 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 294070 T352 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 294070 T352 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 294070 T352 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 294071 T352 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 294071 T352 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 294071 T352 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 294071 T352 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 294074 T353 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@65e92309[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 294076 T352 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 294077 T356 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:38340/zp_tvz/hq collection:collection1 shard:shard2
   [junit4]   2> 294078 T304 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 294078 T304 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 294078 T356 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 294087 T356 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 294089 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 294089 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 294089 T356 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 294089 T356 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 294089 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> ASYNC  NEW_CORE C406 name=collection1 org.apache.solr.core.SolrCore@77515c0f url=http://127.0.0.1:38340/zp_tvz/hq/collection1 node=127.0.0.1:38340_zp_tvz%2Fhq C406_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:38340/zp_tvz/hq, core=collection1, node_name=127.0.0.1:38340_zp_tvz%2Fhq}
   [junit4]   2> 294090 T356 C406 P38340 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:38340/zp_tvz/hq/collection1/
   [junit4]   2> 294090 T356 C406 P38340 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 294091 T356 C406 P38340 oasc.SyncStrategy.syncToMe http://127.0.0.1:38340/zp_tvz/hq/collection1/ has no replicas
   [junit4]   2> 294091 T356 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:38340/zp_tvz/hq/collection1/ shard2
   [junit4]   2> 294091 T356 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 294092 T329 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 294095 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 294097 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 294101 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 294203 T328 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 294203 T338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 294203 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 294248 T356 oasc.ZkController.register We are http://127.0.0.1:38340/zp_tvz/hq/collection1/ and leader is http://127.0.0.1:38340/zp_tvz/hq/collection1/
   [junit4]   2> 294248 T356 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:38340/zp_tvz/hq
   [junit4]   2> 294249 T356 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 294249 T356 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 294249 T356 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 294251 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 294252 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 294252 T356 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 294252 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 294255 T329 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 294256 T329 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:38340/zp_tvz/hq",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38340_zp_tvz%2Fhq",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 294258 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 294344 T304 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 294344 T304 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 294346 T304 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:34468
   [junit4]   2> 294348 T304 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 294348 T304 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 294348 T304 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-004
   [junit4]   2> 294349 T304 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-004/'
   [junit4]   2> 294360 T328 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 294360 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 294360 T338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 294363 T304 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-004/solr.xml
   [junit4]   2> 294406 T304 oasc.CoreContainer.<init> New CoreContainer 434989175
   [junit4]   2> 294407 T304 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-004/]
   [junit4]   2> 294408 T304 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 294409 T304 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 294409 T304 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 294409 T304 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 294410 T304 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 294410 T304 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 294411 T304 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 294411 T304 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 294412 T304 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 294413 T304 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 294413 T304 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 294414 T304 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 294414 T304 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54428/solr
   [junit4]   2> 294415 T304 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 294416 T304 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 294417 T368 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@316569ed name:ZooKeeperConnection Watcher:127.0.0.1:54428 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 294418 T304 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 294420 T304 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 294422 T370 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2628245c name:ZooKeeperConnection Watcher:127.0.0.1:54428/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 294422 T304 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 294431 T304 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 295433 T304 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39069_zp_tvz%2Fhq
   [junit4]   2> 295435 T304 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39069_zp_tvz%2Fhq
   [junit4]   2> 295628 T371 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 295628 T371 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 295630 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 295630 T371 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 295630 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 295630 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 295631 T329 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 295634 T329 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39069/zp_tvz/hq",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:39069_zp_tvz%2Fhq",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 295635 T329 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 295635 T329 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 295637 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 295739 T338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 295739 T370 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 295739 T351 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 295739 T328 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 296630 T371 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 296630 T371 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 296631 T371 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 296631 T371 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 296632 T371 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 296633 T371 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-004/collection1/'
   [junit4]   2> 296633 T371 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 296633 T371 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 296634 T371 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 296665 T371 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 296704 T371 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 296705 T371 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 296713 T371 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 296898 T371 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 296907 T371 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 296909 T371 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 296924 T371 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 296927 T371 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 296930 T371 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 296931 T371 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 296932 T371 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 296932 T371 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 296933 T371 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 296933 T371 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 296933 T371 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 296934 T371 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 296934 T371 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-004/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-001/jetty2/
   [junit4]   2> 296934 T371 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e4bfd69
   [junit4]   2> 296935 T371 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-001/jetty2
   [junit4]   2> 296935 T371 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-001/jetty2/index/
   [junit4]   2> 296935 T371 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 296936 T371 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-BD05FD5B9D125215-001/tempDir-001/jetty2/index
   [junit4]   2> 296936 T371 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=32, maxMergeAtOnceExplicit=41, maxMergedSegmentMB=31.873046875, floorSegmentMB=0.3115234375, forceMergeDeletesPctAllowed=5.064936651571199, segmentsPerTier=43.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3054563728191859
   [junit4]   2> 296937 T371 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RateLimitedDirectoryWrapper(RAMDirectory@55feabfc lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@328b7fd3)),segFN=segments_1,generation=1}
   [junit4]   2> 296938 T371 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 296943 T371 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 296943 T371 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 296943 T371 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 296944 T371 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 296944 T371 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 296944 T371 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 296944 T371 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 296945 T371 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 296945 T371 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 296945 T371 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 296946 T371 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 296946 T371 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 296946 T371 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 296947 T371 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 296947 T371 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 296948 T371 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 296948 T371 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 296950 T371 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 296952 T371 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 296952 T371 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 296953 T371 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=21, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7367731281060768]
   [junit4]   2> 296954 T371 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RateLimitedDirectoryWrapper(RAMDirectory@55feabfc lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@328b7fd3)),segFN=segments_1,generation=1}
   [junit4]   2> 296954 T371 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 296954 T371 oass.SolrIndexSearcher.<init> Opening Searcher@5fa403ef[collection1] main
   [junit4]   2> 296954 T371 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 296955 T371 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 296956 T371 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 296956 T371 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 296956 T371 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 296957 T371 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 296957 T371 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 296957 T371 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 296958 T371 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 296958 T371 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 296961 T372 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5fa403ef[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 296963 T371 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 296964 T375 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:39069/zp_tvz/hq collection:collection1 shard:shard1
   [junit4]   2> 296964 T304 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 296965 T304 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 296965 T375 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 296966 T304 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 296967 T304 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 30000 for each attempt
   [junit4]   2> 296967 T304 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30000
   [junit4]   2> 296968 T304 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 296970 T375 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 296972 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 296972 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 296972 T375 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 296972 T328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2

[...truncated too long message...]

OServerCnxn.java:1081)
   [junit4]   2> 	at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
   [junit4]   2> 	at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
   [junit4]   2> 	at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
   [junit4]   2> 
   [junit4]   2> 1100083 T5458 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception:  java.nio.channels.CancelledKeyException
   [junit4]   2> 	at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
   [junit4]   2> 	at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
   [junit4]   2> 	at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:170)
   [junit4]   2> 	at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:169)
   [junit4]   2> 
   [junit4]   2> 1100085 T5400 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:54600 54600
   [junit4]   2> 1100827 T5455 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:54600 54600
   [junit4]   2> 1100829 T5400 oas.SolrTestCaseJ4.tearDown ###Ending testReconnect
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ZkSolrClientTest -Dtests.method=testReconnect -Dtests.seed=BD05FD5B9D125215 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=zh_HK -Dtests.timezone=Asia/Urumqi -Dtests.file.encoding=UTF-8
   [junit4] ERROR   16.3s J0 | ZkSolrClientTest.testReconnect <<<
   [junit4]    > Throwable #1: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection3
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([BD05FD5B9D125215:64267429DA8B9F6A]:0)
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]    > 	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:329)
   [junit4]    > 	at org.apache.solr.cloud.ZkSolrClientTest.testReconnect(ZkSolrClientTest.java:136)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1102835 T5400 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 1102835 T5400 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=424248011
   [junit4]   2> 1102835 T5400 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@4cb07900
   [junit4]   2> 1102852 T5400 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
   [junit4]   2> 1102852 T5400 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1102853 T5400 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1102853 T5400 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1102853 T5400 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1102853 T5400 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1102854 T5400 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ZkSolrClientTest-BD05FD5B9D125215-001/init-core-data-001/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ZkSolrClientTest-BD05FD5B9D125215-001/init-core-data-001/index;done=false>>]
   [junit4]   2> 1102854 T5400 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ZkSolrClientTest-BD05FD5B9D125215-001/init-core-data-001/index
   [junit4]   2> 1102854 T5400 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ZkSolrClientTest-BD05FD5B9D125215-001/init-core-data-001 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ZkSolrClientTest-BD05FD5B9D125215-001/init-core-data-001;done=false>>]
   [junit4]   2> 1102854 T5400 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ZkSolrClientTest-BD05FD5B9D125215-001/init-core-data-001
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ZkSolrClientTest-BD05FD5B9D125215-001
   [junit4]   2> NOTE: test params are: codec=Lucene42, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=zh_HK, timezone=Asia/Urumqi
   [junit4]   2> NOTE: Linux 3.8.0-41-generic amd64/Oracle Corporation 1.7.0_60-ea (64-bit)/cpus=8,threads=1,free=178295544,total=341049344
   [junit4]   2> NOTE: All tests run in this JVM: [TermsComponentTest, EchoParamsTest, SampleTest, TestFileDictionaryLookup, TestWriterPerf, TestCursorMarkWithoutUniqueKey, SOLR749Test, TestReplicationHandler, TestQueryUtils, StressHdfsTest, FieldMutatingUpdateProcessorTest, UpdateRequestProcessorFactoryTest, ScriptEngineTest, TestFieldSortValues, UnloadDistributedZkTest, SoftAutoCommitTest, TestHighlightDedupGrouping, AliasIntegrationTest, TestCharFilters, TestDefaultSimilarityFactory, TestCloudManagedSchemaAddField, DistributedExpandComponentTest, RecoveryZkTest, TestSort, TestSolrXmlPersistor, DirectUpdateHandlerOptimizeTest, TestSimpleQParserPlugin, CursorPagingTest, TestBlendedInfixSuggestions, DistributedQueryComponentCustomSortTest, TestJmxMonitoredMap, BasicFunctionalityTest, SuggesterFSTTest, TestConfig, EnumFieldTest, QueryEqualityTest, OverseerRolesTest, TestNonDefinedSimilarityFactory, DistributedQueryElevationComponentTest, TestRemoteStreaming, UpdateParamsTest, TestSolrIndexConfig, TestNonNRTOpen, MultiThreadedOCPTest, TestLuceneMatchVersion, ReturnFieldsTest, TestStressVersions, TestExceedMaxTermLength, TestDistributedMissingSort, TestSearchPerf, RAMDirectoryFactoryTest, SyncSliceTest, InfoHandlerTest, SolrCoreTest, FastVectorHighlighterTest, DistributedSuggestComponentTest, TestElisionMultitermQuery, ShardRoutingTest, TestCollationFieldDocValues, SearchHandlerTest, TestReversedWildcardFilterFactory, TestDocSet, TestLFUCache, BJQParserTest, ReplicationFactorTest, HdfsChaosMonkeySafeLeaderTest, TestJoin, TestRandomMergePolicy, TestUtils, OverseerCollectionProcessorTest, TestInitQParser, TestGroupingSearch, QueryElevationComponentTest, TestLMJelinekMercerSimilarityFactory, DistribDocExpirationUpdateProcessorTest, TestFastLRUCache, TestSolrDeletionPolicy1, HdfsRecoveryZkTest, ClusterStateUpdateTest, TermVectorComponentTest, AbstractAnalyticsFacetTest, TestFieldResource, CSVRequestHandlerTest, TestUniqueKeyFieldResource, StatelessScriptUpdateProcessorFactoryTest, TimeZoneUtilsTest, TestOrdValues, TestShortCircuitedRequests, RegexBoostProcessorTest, TestPseudoReturnFields, AutoCommitTest, DocValuesTest, OverseerTest, TestSystemIdResolver, DistributedTermsComponentTest, SuggesterTest, SpatialFilterTest, UUIDFieldTest, BasicDistributedZkTest, OpenCloseCoreStressTest, LeaderElectionIntegrationTest, FullSolrCloudDistribCmdsTest, LeaderElectionTest, ZkSolrClientTest]
   [junit4] Completed on J0 in 23.31s, 5 tests, 1 error <<< FAILURES!

[...truncated 633 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1296: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:920: There were test failures: 406 suites, 1688 tests, 2 errors, 1 failure, 36 ignored (15 assumptions)

Total time: 61 minutes 38 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_60-ea-b15 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure