You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/01/13 12:03:00 UTC

[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.7.0_60-ea-b02) - Build # 8975 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/8975/
Java: 64bit/jdk1.7.0_60-ea-b02 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest:     1) Thread[id=1607, name=Timer-0, state=WAITING, group=TGRP-BasicDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:503)         at java.util.TimerThread.mainLoop(Timer.java:526)         at java.util.TimerThread.run(Timer.java:505)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest: 
   1) Thread[id=1607, name=Timer-0, state=WAITING, group=TGRP-BasicDistributedZkTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:503)
        at java.util.TimerThread.mainLoop(Timer.java:526)
        at java.util.TimerThread.run(Timer.java:505)
	at __randomizedtesting.SeedInfo.seed([EFDFCA0C1F266D7D]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=1607, name=Timer-0, state=WAITING, group=TGRP-BasicDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:503)         at java.util.TimerThread.mainLoop(Timer.java:526)         at java.util.TimerThread.run(Timer.java:505)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=1607, name=Timer-0, state=WAITING, group=TGRP-BasicDistributedZkTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:503)
        at java.util.TimerThread.mainLoop(Timer.java:526)
        at java.util.TimerThread.run(Timer.java:505)
	at __randomizedtesting.SeedInfo.seed([EFDFCA0C1F266D7D]:0)




Build Log:
[...truncated 10489 lines...]
   [junit4] Suite: org.apache.solr.cloud.BasicDistributedZkTest
   [junit4]   2> 733121 T1260 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 733127 T1260 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZkTest-1389610757147
   [junit4]   2> 733128 T1260 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 733129 T1261 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 733229 T1260 oasc.ZkTestServer.run start zk server on port:57401
   [junit4]   2> 733230 T1260 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 735433 T1264 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 2199ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
   [junit4]   2> 735435 T1267 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d84ca38 name:ZooKeeperConnection Watcher:127.0.0.1:57401 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 735435 T1260 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 735435 T1260 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 735441 T1260 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 735442 T1269 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2bc75807 name:ZooKeeperConnection Watcher:127.0.0.1:57401/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 735442 T1260 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 735442 T1260 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 735445 T1260 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 735447 T1260 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 735449 T1260 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 735451 T1260 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 735452 T1260 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 737693 T1264 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 2237ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
   [junit4]   2> 737695 T1260 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 737695 T1260 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 737701 T1260 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 737702 T1260 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 737707 T1260 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 737708 T1260 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 737712 T1260 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 737713 T1260 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 737718 T1260 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 737718 T1260 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 737723 T1260 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 737723 T1260 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 737729 T1260 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 737729 T1260 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 737733 T1260 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 737734 T1260 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 737740 T1260 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 737740 T1260 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 737745 T1260 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 737745 T1260 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 737831 T1260 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 737836 T1260 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:45686
   [junit4]   2> 737836 T1260 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 737836 T1260 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 737837 T1260 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1389610761772
   [junit4]   2> 737837 T1260 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1389610761772/'
   [junit4]   2> 737847 T1260 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1389610761772/solr.xml
   [junit4]   2> 737876 T1260 oasc.CoreContainer.<init> New CoreContainer 230411391
   [junit4]   2> 737876 T1260 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1389610761772/]
   [junit4]   2> 737877 T1260 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 737877 T1260 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 737877 T1260 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 737877 T1260 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 737878 T1260 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 737878 T1260 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 737878 T1260 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 737878 T1260 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 737879 T1260 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 737881 T1260 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 737881 T1260 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 737881 T1260 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 737882 T1260 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57401/solr
   [junit4]   2> 737882 T1260 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 737883 T1260 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 737885 T1280 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@246eace4 name:ZooKeeperConnection Watcher:127.0.0.1:57401 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 737886 T1260 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 737888 T1260 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 737889 T1282 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@22a1453d name:ZooKeeperConnection Watcher:127.0.0.1:57401/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 737889 T1260 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 737891 T1260 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 737893 T1260 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 737896 T1260 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 737897 T1260 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45686_
   [junit4]   2> 737897 T1260 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45686_
   [junit4]   2> 737899 T1260 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 737901 T1260 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 737908 T1260 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 737910 T1260 oasc.Overseer.start Overseer (id=91069530580647939-127.0.0.1:45686_-n_0000000000) starting
   [junit4]   2> 737913 T1260 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 737915 T1284 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 737915 T1260 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 737918 T1260 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 737919 T1260 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 737921 T1283 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 737923 T1285 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 737923 T1285 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 737924 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 737924 T1285 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 737925 T1283 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 737926 T1283 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:45686",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:45686_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 737926 T1283 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 737926 T1283 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 737928 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 737929 T1282 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 738925 T1285 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 738925 T1285 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1389610761772/collection1
   [junit4]   2> 738925 T1285 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 738927 T1285 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 738927 T1285 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 738929 T1285 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1389610761772/collection1/'
   [junit4]   2> 738929 T1285 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1389610761772/collection1/lib/classes/' to classloader
   [junit4]   2> 738930 T1285 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1389610761772/collection1/lib/README' to classloader
   [junit4]   2> 738976 T1285 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 738999 T1285 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 739000 T1285 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 739005 T1285 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 739225 T1285 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 739225 T1285 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 739225 T1285 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 739229 T1285 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 739231 T1285 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 739245 T1285 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 739248 T1285 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 739251 T1285 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 739252 T1285 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 739252 T1285 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 739252 T1285 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 739253 T1285 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 739253 T1285 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 739253 T1285 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 739253 T1285 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1389610761772/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1389610757147/control/data/
   [junit4]   2> 739254 T1285 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@81ebbfc
   [junit4]   2> 739254 T1285 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1389610757147/control/data
   [junit4]   2> 739254 T1285 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1389610757147/control/data/index/
   [junit4]   2> 739255 T1285 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1389610757147/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 739255 T1285 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1389610757147/control/data/index
   [junit4]   2> 739255 T1285 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=46, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 739256 T1285 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@127b59b5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@192eaf04),segFN=segments_1,generation=1}
   [junit4]   2> 739256 T1285 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 739259 T1285 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 739260 T1285 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 739260 T1285 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 739260 T1285 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 739260 T1285 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 739260 T1285 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 739261 T1285 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 739261 T1285 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 739261 T1285 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 739261 T1285 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 739262 T1285 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 739262 T1285 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 739262 T1285 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 739262 T1285 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 739263 T1285 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 739263 T1285 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 739265 T1285 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 739267 T1285 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 739267 T1285 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 739267 T1285 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=14, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=44.3193359375, floorSegmentMB=1.5751953125, forceMergeDeletesPctAllowed=20.74270045064597, segmentsPerTier=11.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1651870914398942
   [junit4]   2> 739267 T1285 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@127b59b5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@192eaf04),segFN=segments_1,generation=1}
   [junit4]   2> 739268 T1285 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 739268 T1285 oass.SolrIndexSearcher.<init> Opening Searcher@4dfbb8c8 main
   [junit4]   2> 739270 T1286 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4dfbb8c8 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 739270 T1285 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 739271 T1260 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 739272 T1260 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 739271 T1289 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:45686 collection:control_collection shard:shard1
   [junit4]   2> 739273 T1260 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 739274 T1289 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 739274 T1291 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a4497b8 name:ZooKeeperConnection Watcher:127.0.0.1:57401/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 739275 T1260 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 739277 T1260 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 739280 T1260 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 739282 T1289 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 739284 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 739285 T1289 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 739285 T1289 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C11177 name=collection1 org.apache.solr.core.SolrCore@fa9a836 url=http://127.0.0.1:45686/collection1 node=127.0.0.1:45686_ C11177_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:45686, core=collection1, node_name=127.0.0.1:45686_}
   [junit4]   2> 739285 T1289 C11177 P45686 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:45686/collection1/
   [junit4]   2> 739286 T1289 C11177 P45686 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 739286 T1289 C11177 P45686 oasc.SyncStrategy.syncToMe http://127.0.0.1:45686/collection1/ has no replicas
   [junit4]   2> 739286 T1283 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 739286 T1289 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:45686/collection1/ shard1
   [junit4]   2> 739286 T1289 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 739291 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 739298 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 739301 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 739352 T1260 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 739354 T1260 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60276
   [junit4]   2> 739355 T1260 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 739356 T1260 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 739356 T1260 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1389610763300
   [junit4]   2> 739357 T1260 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1389610763300/'
   [junit4]   2> 739376 T1260 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1389610763300/solr.xml
   [junit4]   2> 739404 T1282 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 739404 T1291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 739406 T1260 oasc.CoreContainer.<init> New CoreContainer 1429573629
   [junit4]   2> 739406 T1260 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1389610763300/]
   [junit4]   2> 739407 T1260 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 739407 T1260 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 739407 T1260 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 739408 T1260 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 739408 T1260 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 739408 T1260 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 739408 T1260 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 739409 T1260 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 739409 T1260 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 739411 T1260 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 739411 T1260 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 739412 T1260 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 739412 T1260 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57401/solr
   [junit4]   2> 739412 T1260 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 739413 T1260 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 739415 T1302 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12bf2df6 name:ZooKeeperConnection Watcher:127.0.0.1:57401 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 739416 T1260 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 739419 T1260 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 739420 T1304 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4124b06b name:ZooKeeperConnection Watcher:127.0.0.1:57401/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 739421 T1260 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 739429 T1260 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 739449 T1289 oasc.ZkController.register We are http://127.0.0.1:45686/collection1/ and leader is http://127.0.0.1:45686/collection1/
   [junit4]   2> 739449 T1289 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45686
   [junit4]   2> 739449 T1289 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 739449 T1289 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 739449 T1289 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 739451 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 739452 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 739452 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 739452 T1289 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 739455 T1283 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 739456 T1283 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:45686",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:45686_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 739461 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 739564 T1291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 739564 T1304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 739564 T1282 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 740436 T1260 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60276_
   [junit4]   2> 740458 T1260 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60276_
   [junit4]   2> 740482 T1282 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 740482 T1291 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 740482 T1304 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 740505 T1305 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 740505 T1305 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 740533 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 740533 T1305 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 740533 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 740534 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 740537 T1283 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 740538 T1283 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:60276",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:60276_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 740539 T1283 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 740539 T1283 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 740547 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 740651 T1282 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 740651 T1291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 740651 T1304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 741534 T1305 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 741534 T1305 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1389610763300/collection1
   [junit4]   2> 741534 T1305 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 741535 T1305 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 741535 T1305 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 741536 T1305 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1389610763300/collection1/'
   [junit4]   2> 741537 T1305 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1389610763300/collection1/lib/classes/' to classloader
   [junit4]   2> 741537 T1305 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1389610763300/collection1/lib/README' to classloader
   [junit4]   2> 741579 T1305 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 741596 T1305 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 741598 T1305 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 741603 T1305 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 741835 T1305 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 741836 T1305 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 741836 T1305 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 741840 T1305 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 741841 T1305 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 741856 T1305 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 741860 T1305 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 741864 T1305 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 741866 T1305 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 741866 T1305 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 741867 T1305 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 741868 T1305 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 741868 T1305 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 741869 T1305 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 741869 T1305 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1389610763300/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1389610757147/jetty1/
   [junit4]   2> 741869 T1305 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@81ebbfc
   [junit4]   2> 741870 T1305 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1389610757147/jetty1
   [junit4]   2> 741871 T1305 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1389610757147/jetty1/index/
   [junit4]   2> 741871 T1305 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1389610757147/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 741871 T1305 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1389610757147/jetty1/index
   [junit4]   2> 741872 T1305 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=46, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 741873 T1305 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@aaab288 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@61ec09a0),segFN=segments_1,generation=1}
   [junit4]   2> 741873 T1305 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 741879 T1305 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 741880 T1305 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 741880 T1305 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 741881 T1305 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 741881 T1305 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 741881 T1305 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 741882 T1305 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 741883 T1305 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 741883 T1305 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 741884 T1305 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 741885 T1305 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 741885 T1305 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 741885 T1305 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 741886 T1305 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 741887 T1305 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 741888 T1305 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 741891 T1305 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 741894 T1305 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 741895 T1305 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 741896 T1305 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=14, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=44.3193359375, floorSegmentMB=1.5751953125, forceMergeDeletesPctAllowed=20.74270045064597, segmentsPerTier=11.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1651870914398942
   [junit4]   2> 741896 T1305 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@aaab288 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@61ec09a0),segFN=segments_1,generation=1}
   [junit4]   2> 741897 T1305 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 741897 T1305 oass.SolrIndexSearcher.<init> Opening Searcher@6be6115a main
   [junit4]   2> 741907 T1306 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6be6115a main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 741909 T1305 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 741910 T1260 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 741910 T1260 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 741910 T1309 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:60276 collection:collection1 shard:shard2
   [junit4]   2> 741912 T1309 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 741921 T1309 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 741924 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 741924 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 741925 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 741925 T1309 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 741925 T1309 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C11178 name=collection1 org.apache.solr.core.SolrCore@1d43b243 url=http://127.0.0.1:60276/collection1 node=127.0.0.1:60276_ C11178_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:60276, core=collection1, node_name=127.0.0.1:60276_}
   [junit4]   2> 741925 T1309 C11178 P60276 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:60276/collection1/
   [junit4]   2> 741926 T1309 C11178 P60276 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 741926 T1309 C11178 P60276 oasc.SyncStrategy.syncToMe http://127.0.0.1:60276/collection1/ has no replicas
   [junit4]   2> 741926 T1309 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:60276/collection1/ shard2
   [junit4]   2> 741927 T1309 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 741927 T1283 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 741933 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 741936 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 741940 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 741991 T1260 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 741994 T1260 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50746
   [junit4]   2> 741994 T1260 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 741995 T1260 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 741995 T1260 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1389610765930
   [junit4]   2> 741996 T1260 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1389610765930/'
   [junit4]   2> 742007 T1260 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1389610765930/solr.xml
   [junit4]   2> 742026 T1260 oasc.CoreContainer.<init> New CoreContainer 1422007954
   [junit4]   2> 742026 T1260 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1389610765930/]
   [junit4]   2> 742027 T1260 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 742027 T1260 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 742027 T1260 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 742027 T1260 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 742028 T1260 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 742028 T1260 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 742028 T1260 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 742028 T1260 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 742028 T1260 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 742031 T1260 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 742031 T1260 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 742031 T1260 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 742032 T1260 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57401/solr
   [junit4]   2> 742032 T1260 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 742033 T1260 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 742036 T1320 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14652eb7 name:ZooKeeperConnection Watcher:127.0.0.1:57401 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 742036 T1260 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 742040 T1260 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 742043 T1322 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@55c75e7b name:ZooKeeperConnection Watcher:127.0.0.1:57401/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 742043 T1260 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 742047 T1282 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 742047 T1304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 742047 T1291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 742053 T1260 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 742087 T1309 oasc.ZkController.register We are http://127.0.0.1:60276/collection1/ and leader is http://127.0.0.1:60276/collection1/
   [junit4]   2> 742087 T1309 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60276
   [junit4]   2> 742088 T1309 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 742088 T1309 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 742088 T1309 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 742090 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 742090 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 742090 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 742090 T1309 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 742093 T1283 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 742095 T1283 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:60276",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:60276_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 742099 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 742203 T1282 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 742203 T1322 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 742203 T1304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 742203 T1291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 743058 T1260 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50746_
   [junit4]   2> 743060 T1260 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50746_
   [junit4]   2> 743064 T1291 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 743065 T1282 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 743065 T1304 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 743065 T1322 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 743085 T1323 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 743085 T1323 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 743087 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 743088 T1323 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 743088 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 743088 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 743091 T1283 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 743093 T1283 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50746",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50746_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 743093 T1283 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 743093 T1283 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 743098 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 743201 T1282 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 743202 T1291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 743202 T1304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 743201 T1322 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 744088 T1323 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 744089 T1323 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1389610765930/collection1
   [junit4]   2> 744089 T1323 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 744090 T1323 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 744090 T1323 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 744091 T1323 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1389610765930/collection1/'
   [junit4]   2> 744092 T1323 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1389610765930/collection1/lib/classes/' to classloader
   [junit4]   2> 744092 T1323 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1389610765930/collection1/lib/README' to classloader
   [junit4]   2> 744131 T1323 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 744165 T1323 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 744167 T1323 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 744173 T1323 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 744421 T1323 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 744421 T1323 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 744422 T1323 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 744426 T1323 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 744428 T1323 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 744446 T1323 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 744450 T1323 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 744453 T1323 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 744454 T1323 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 744454 T1323 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 744455 T1323 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 744456 T1323 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 744456 T1323 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 744456 T1323 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 744456 T1323 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1389610765930/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1389610757147/jetty2/
   [junit4]   2> 744457 T1323 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@81ebbfc
   [junit4]   2> 744457 T1323 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1389610757147/jetty2
   [junit4]   2> 744457 T1323 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1389610757147/jetty2/index/
   [junit4]   2> 744458 T1323 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1389610757147/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 744458 T1323 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1389610757147/jetty2/index
   [junit4]   2> 744458 T1323 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=46, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 744459 T1323 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@368a3595 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6de072bb),segFN=segments_1,generation=1}
   [junit4]   2> 744459 T1323 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 744464 T1323 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 744464 T1323 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 744464 T1323 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 744465 T1323 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 744465 T1323 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 744465 T1323 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 744465 T1323 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 744465 T1323 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 744466 T1323 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 744466 T1323 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 744467 T1323 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 744467 T1323 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 744468 T1323 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 744468 T1323 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 744468 T1323 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 744469 T1323 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 744471 T1323 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 744473 T1323 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 744473 T1323 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 744474 T1323 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=14, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=44.3193359375, floorSegmentMB=1.5751953125, forceMergeDeletesPctAllowed=20.74270045064597, segmentsPerTier=11.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1651870914398942
   [junit4]   2> 744474 T1323 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@368a3595 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6de072bb),segFN=segments_1,generation=1}
   [junit4]   2> 744474 T1323 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 744475 T1323 oass.SolrIndexSearcher.<init> Opening Searcher@2daa732b main
   [junit4]   2> 744477 T1324 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2daa732b main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 744480 T1323 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 744480 T1260 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 744481 T1260 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 744482 T1327 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50746 collection:collection1 shard:shard1
   [junit4]   2> 744483 T1327 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 744525 T1327 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 744529 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 744529 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 744529 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 744530 T1327 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 744530 T1327 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C11179 name=collection1 org.apache.solr.core.SolrCore@51663e97 url=http://127.0.0.1:50746/collection1 node=127.0.0.1:50746_ C11179_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:50746, core=collection1, node_name=127.0.0.1:50746_}
   [junit4]   2> 744530 T1327 C11179 P50746 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50746/collection1/
   [junit4]   2> 744531 T1327 C11179 P50746 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 744531 T1283 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 744531 T1327 C11179 P50746 oasc.SyncStrategy.syncToMe http://127.0.0.1:50746/collection1/ has no replicas
   [junit4]   2> 744531 T1327 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50746/collection1/ shard1
   [junit4]   2> 744532 T1327 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 744534 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 744537 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 744542 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 744558 T1260 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 744560 T1260 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:51393
   [junit4]   2> 744561 T1260 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 744561 T1260 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 744561 T1260 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1389610768500
   [junit4]   2> 744561 T1260 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1389610768500/'
   [junit4]   2> 744575 T1260 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1389610768500/solr.xml
   [junit4]   2> 744604 T1260 oasc.CoreContainer.<init> New CoreContainer 1812788606
   [junit4]   2> 744604 T1260 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1389610768500/]
   [junit4]   2> 744605 T1260 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 744605 T1260 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 744605 T1260 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 744606 T1260 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 744606 T1260 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 744606 T1260 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 744606 T1260 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 744607 T1260 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 744607 T1260 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 744609 T1260 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 744610 T1260 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 744610 T1260 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 744610 T1260 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57401/solr
   [junit4]   2> 744611 T1260 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 744611 T1260 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 744613 T1338 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@46c95a9d name:ZooKeeperConnection Watcher:127.0.0.1:57401 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 744615 T1260 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 744619 T1260 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 744621 T1340 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@34e86947 name:ZooKeeperConnection Watcher:127.0.0.1:57401/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 744621 T1260 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 744628 T1260 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 744645 T1322 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 744645 T1304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 744645 T1291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 744645 T1282 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 744645 T1340 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 744688 T1327 oasc.ZkController.register We are http://127.0.0.1:50746/collection1/ and leader is http://127.0.0.1:50746/collection1/
   [junit4]   2> 744689 T1327 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50746
   [junit4]   2> 744689 T1327 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 744689 T1327 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 744690 T1327 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 744691 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 744692 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 744692 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 744692 T1327 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 744695 T1283 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 744696 T1283 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50746",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50746_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 744719 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 744824 T1282 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 744824 T1304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 744824 T1340 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 744824 T1291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 744824 T1322 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 745633 T1260 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51393_
   [junit4]   2> 745635 T1260 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51393_
   [junit4]   2> 745637 T1291 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 745638 T1322 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 745638 T1340 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 745638 T1304 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 745638 T1282 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 745643 T1341 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 745644 T1341 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 745645 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 745645 T1341 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 745645 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 745645 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 745647 T1283 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 745647 T1283 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51393",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51393_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 745648 T1283 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 745648 T1283 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 745650 T1282 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 745753 T1282 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 745754 T1340 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 745753 T1322 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 745753 T1291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 745753 T1304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 746645 T1341 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 746645 T1341 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1389610768500/collection1
   [junit4]   2> 746645 T1341 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 746646 T1341 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 746647 T1341 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 746648 T1341 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1389610768500/collection1/'
   [junit4]   2> 746648 T1341 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1389610768500/collection1/lib/classes/' to classloader
   [junit4]   2> 746649 T1341 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1389610768500/collection1/lib/README' to classloader
   [junit4]   2> 746673 T1341 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 746695 T1341 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 746697 T1341 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 746703 T1341 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 746959 T1341 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 746960 T1341 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 746961 T1341 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 746974 T1341 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 746976 T1341 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 746987 T1341 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 746990 T1341 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 746993 T1341 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 746994 T1341 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 746994 T1341 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 746994 T1341 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 746995 T1341 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 746995 T1341 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 746995 T1341 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 746996 T1341 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1389610768500/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1389610757147/jetty3/
   [junit4]   2> 746996 T1341 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@81ebbfc
   [junit4]   2> 746997 T1341 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1389610757147/jetty3
   [junit4]   2> 746997 T1341 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1389610757147/jetty3/index/
   [junit4]   2> 746997 T1341 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1389610757147/jetty3/index' doesn't exist. Creating new index...
   [junit4]   2> 746998 T1341 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1389610757147/jetty3/index
   [junit4]   2> 746998 T1341 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=46, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 746999 T1341 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@691d32b5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@26a15325),segFN=segments_1,generation=1}
   [junit4]   2> 746999 T1341 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 

[...truncated too long message...]

1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:91)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:136)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 820486 T1359 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection3/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:91)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:136)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 820487 T1359 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 820487 T1359 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 820487 T1359 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:91)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:136)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 87637 T1259 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> 107714 T1259 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest: 
   [junit4]   2> 	   1) Thread[id=1607, name=Timer-0, state=WAITING, group=TGRP-BasicDistributedZkTest]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.lang.Object.wait(Object.java:503)
   [junit4]   2> 	        at java.util.TimerThread.mainLoop(Timer.java:526)
   [junit4]   2> 	        at java.util.TimerThread.run(Timer.java:505)
   [junit4]   2> 107714 T1259 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
   [junit4]   2> 	   1) Thread[id=1607, name=Timer-0, state=WAITING, group=TGRP-BasicDistributedZkTest]
   [junit4]   2> 110718 T1259 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
   [junit4]   2> 	   1) Thread[id=1607, name=Timer-0, state=WAITING, group=TGRP-BasicDistributedZkTest]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.lang.Object.wait(Object.java:503)
   [junit4]   2> 	        at java.util.TimerThread.mainLoop(Timer.java:526)
   [junit4]   2> 	        at java.util.TimerThread.run(Timer.java:505)
   [junit4]   2> NOTE: test params are: codec=Lucene42, sim=DefaultSimilarity, locale=sr_ME_#Latn, timezone=Africa/Johannesburg
   [junit4]   2> NOTE: Linux 3.8.0-35-generic amd64/Oracle Corporation 1.7.0_60-ea (64-bit)/cpus=8,threads=2,free=396953104,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [MBeansHandlerTest, TestInfoStreamLogging, UpdateParamsTest, ExternalFileFieldSortTest, LegacyHTMLStripCharFilterTest, CacheHeaderTest, TestPerFieldSimilarity, SoftAutoCommitTest, TestClassNameShortening, DistributedSpellCheckComponentTest, TestQueryUtils, TestRealTimeGet, TestLFUCache, TestReloadAndDeleteDocs, AutoCommitTest, TestPHPSerializedResponseWriter, RegexBoostProcessorTest, CoreAdminCreateDiscoverTest, TestJoin, TestPostingsSolrHighlighter, JsonLoaderTest, ResponseLogComponentTest, TestElisionMultitermQuery, HdfsBasicDistributedZk2Test, InfoHandlerTest, WordBreakSolrSpellCheckerTest, TestLMDirichletSimilarityFactory, ClusterStateTest, SOLR749Test, ReturnFieldsTest, SynonymTokenizerTest, TestFieldTypeCollectionResource, TestValueSourceCache, SpatialFilterTest, TestCopyFieldCollectionResource, DocumentBuilderTest, RecoveryZkTest, BlockCacheTest, ZkControllerTest, TestCollationFieldDocValues, TestSolrXmlPersistor, TestSchemaNameResource, TestConfig, UnloadDistributedZkTest, DOMUtilTest, ParsingFieldUpdateProcessorsTest, SuggesterFSTTest, TestCharFilters, DistribCursorPagingTest, TestDocSet, TestCollapseQParserPlugin, HdfsDirectoryTest, TestSolrXMLSerializer, BasicFunctionalityTest, URLClassifyProcessorTest, TestRandomDVFaceting, SuggesterTest, TestStressVersions, SystemInfoHandlerTest, DistributedTermsComponentTest, TestFunctionQuery, BasicDistributedZkTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=BasicDistributedZkTest -Dtests.seed=EFDFCA0C1F266D7D -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sr_ME_#Latn -Dtests.timezone=Africa/Johannesburg -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J0 | BasicDistributedZkTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest: 
   [junit4]    >    1) Thread[id=1607, name=Timer-0, state=WAITING, group=TGRP-BasicDistributedZkTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at java.lang.Object.wait(Object.java:503)
   [junit4]    >         at java.util.TimerThread.mainLoop(Timer.java:526)
   [junit4]    >         at java.util.TimerThread.run(Timer.java:505)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([EFDFCA0C1F266D7D]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=1607, name=Timer-0, state=WAITING, group=TGRP-BasicDistributedZkTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at java.lang.Object.wait(Object.java:503)
   [junit4]    >         at java.util.TimerThread.mainLoop(Timer.java:526)
   [junit4]    >         at java.util.TimerThread.run(Timer.java:505)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([EFDFCA0C1F266D7D]:0)
   [junit4] Completed on J0 in 110.77s, 1 test, 2 errors <<< FAILURES!

[...truncated 610 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:459: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:439: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1307: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:940: There were test failures: 352 suites, 1537 tests, 2 suite-level errors, 845 ignored (1 assumption)

Total time: 37 minutes 19 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_60-ea-b02 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure