You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/01/10 14:10:05 UTC

[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_45) - Build # 9040 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9040/
Java: 32bit/jdk1.7.0_45 -server -XX:+UseSerialGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest:     1) Thread[id=1061, name=Timer-0, state=WAITING, group=TGRP-BasicDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:503)         at java.util.TimerThread.mainLoop(Timer.java:526)         at java.util.TimerThread.run(Timer.java:505)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest: 
   1) Thread[id=1061, name=Timer-0, state=WAITING, group=TGRP-BasicDistributedZkTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:503)
        at java.util.TimerThread.mainLoop(Timer.java:526)
        at java.util.TimerThread.run(Timer.java:505)
	at __randomizedtesting.SeedInfo.seed([113AA8383E83B11D]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=1061, name=Timer-0, state=WAITING, group=TGRP-BasicDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:503)         at java.util.TimerThread.mainLoop(Timer.java:526)         at java.util.TimerThread.run(Timer.java:505)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=1061, name=Timer-0, state=WAITING, group=TGRP-BasicDistributedZkTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:503)
        at java.util.TimerThread.mainLoop(Timer.java:526)
        at java.util.TimerThread.run(Timer.java:505)
	at __randomizedtesting.SeedInfo.seed([113AA8383E83B11D]:0)




Build Log:
[...truncated 10091 lines...]
   [junit4] Suite: org.apache.solr.cloud.BasicDistributedZkTest
   [junit4]   2> 101030 T628 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /yi_oj/
   [junit4]   2> 101033 T628 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZkTest-1389358617190
   [junit4]   2> 101034 T628 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 101034 T629 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 101134 T628 oasc.ZkTestServer.run start zk server on port:38396
   [junit4]   2> 101135 T628 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 101235 T635 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19b6f08 name:ZooKeeperConnection Watcher:127.0.0.1:38396 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 101235 T628 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 101236 T628 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 101243 T628 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 101243 T630 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1437c3a12680000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 101245 T637 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1546c6e name:ZooKeeperConnection Watcher:127.0.0.1:38396/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 101246 T628 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 101246 T628 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 101249 T628 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 101252 T628 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 101255 T628 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 101257 T628 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 101258 T628 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 101265 T628 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 101266 T628 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 101270 T628 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 101271 T628 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 101274 T628 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 101275 T628 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 101278 T628 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 101279 T628 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 101282 T628 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 101283 T628 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 101286 T628 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 101287 T628 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 101290 T628 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 101291 T628 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 101294 T628 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 101295 T628 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 101298 T628 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 101299 T628 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 101302 T628 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 101302 T628 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 101375 T628 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 101378 T628 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:37400
   [junit4]   2> 101379 T628 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 101379 T628 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 101379 T628 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1389358617462
   [junit4]   2> 101380 T628 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1389358617462/'
   [junit4]   2> 101395 T628 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1389358617462/solr.xml
   [junit4]   2> 101431 T628 oasc.CoreContainer.<init> New CoreContainer 5211182
   [junit4]   2> 101431 T628 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1389358617462/]
   [junit4]   2> 101432 T628 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 101432 T628 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 101433 T628 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 101433 T628 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 101433 T628 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 101434 T628 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 101434 T628 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 101434 T628 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 101434 T628 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 101437 T628 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 101437 T628 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 101438 T628 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 101438 T628 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38396/solr
   [junit4]   2> 101438 T628 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 101439 T628 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 101442 T648 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6f219c name:ZooKeeperConnection Watcher:127.0.0.1:38396 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 101442 T628 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 101445 T628 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 101446 T650 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b80f78 name:ZooKeeperConnection Watcher:127.0.0.1:38396/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 101446 T628 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 101448 T628 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 101452 T628 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 101455 T628 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 101457 T628 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37400_yi_oj
   [junit4]   2> 101458 T628 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37400_yi_oj
   [junit4]   2> 101461 T628 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 101464 T628 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 101469 T628 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 101574 T628 oasc.Overseer.start Overseer (id=91053006336294915-127.0.0.1:37400_yi_oj-n_0000000000) starting
   [junit4]   2> 101579 T628 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 101585 T652 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 101586 T628 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 101590 T628 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 101592 T628 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 101596 T651 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 101600 T653 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 101600 T653 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 101602 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 101602 T653 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 101606 T651 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 101607 T651 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:37400/yi_oj",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:37400_yi_oj",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 101607 T651 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 101608 T651 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 101611 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 101612 T650 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 102603 T653 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 102604 T653 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1389358617462/collection1
   [junit4]   2> 102604 T653 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 102606 T653 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 102606 T653 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 102608 T653 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1389358617462/collection1/'
   [junit4]   2> 102609 T653 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1389358617462/collection1/lib/classes/' to classloader
   [junit4]   2> 102609 T653 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1389358617462/collection1/lib/README' to classloader
   [junit4]   2> 102644 T653 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 102697 T653 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 102698 T653 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 102706 T653 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 103042 T653 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 103048 T653 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 103051 T653 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 103069 T653 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 103073 T653 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 103078 T653 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 103080 T653 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 103080 T653 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 103081 T653 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 103082 T653 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 103083 T653 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 103083 T653 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 103083 T653 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1389358617462/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1389358617189/control/data/
   [junit4]   2> 103084 T653 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5f6fc9
   [junit4]   2> 103084 T653 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1389358617189/control/data
   [junit4]   2> 103085 T653 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1389358617189/control/data/index/
   [junit4]   2> 103085 T653 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1389358617189/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 103087 T653 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1389358617189/control/data/index
   [junit4]   2> 103088 T653 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1919050342, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 103090 T653 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/index5769757698tmp lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4bc04c),segFN=segments_1,generation=1}
   [junit4]   2> 103090 T653 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 103095 T653 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 103095 T653 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 103096 T653 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 103096 T653 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 103096 T653 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 103097 T653 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 103097 T653 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 103097 T653 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 103098 T653 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 103098 T653 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 103098 T653 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 103099 T653 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 103099 T653 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 103099 T653 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 103100 T653 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 103100 T653 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 103101 T653 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 103104 T653 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 103107 T653 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 103108 T653 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 103108 T653 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=956245469, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 103110 T653 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/index5769757698tmp lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4bc04c),segFN=segments_1,generation=1}
   [junit4]   2> 103110 T653 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 103110 T653 oass.SolrIndexSearcher.<init> Opening Searcher@1266794 main
   [junit4]   2> 103113 T654 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1266794 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 103114 T653 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 103115 T628 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 103115 T628 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 103115 T657 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:37400/yi_oj collection:control_collection shard:shard1
   [junit4]   2> 103117 T628 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 103117 T657 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 103118 T659 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@194aa8 name:ZooKeeperConnection Watcher:127.0.0.1:38396/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 103119 T628 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 103121 T628 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 103125 T628 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 103127 T657 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 103129 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 103130 T657 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 103130 T657 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C233 name=collection1 org.apache.solr.core.SolrCore@1a5000 url=http://127.0.0.1:37400/yi_oj/collection1 node=127.0.0.1:37400_yi_oj C233_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:37400/yi_oj, core=collection1, node_name=127.0.0.1:37400_yi_oj}
   [junit4]   2> 103131 T657 C233 P37400 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:37400/yi_oj/collection1/
   [junit4]   2> 103131 T657 C233 P37400 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 103131 T651 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 103131 T657 C233 P37400 oasc.SyncStrategy.syncToMe http://127.0.0.1:37400/yi_oj/collection1/ has no replicas
   [junit4]   2> 103132 T657 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:37400/yi_oj/collection1/ shard1
   [junit4]   2> 103132 T657 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 103135 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 103136 T650 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 103136 T659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 103211 T628 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 103213 T628 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:56128
   [junit4]   2> 103214 T628 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 103214 T628 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 103214 T628 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1389358619282
   [junit4]   2> 103214 T628 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1389358619282/'
   [junit4]   2> 103232 T628 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1389358619282/solr.xml
   [junit4]   2> 103269 T628 oasc.CoreContainer.<init> New CoreContainer 25514881
   [junit4]   2> 103269 T628 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1389358619282/]
   [junit4]   2> 103270 T628 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 103271 T628 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 103271 T628 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 103271 T651 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 103271 T628 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 103272 T628 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 103272 T628 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 103273 T628 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 103273 T628 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 103273 T628 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 103275 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 103276 T628 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 103276 T628 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 103277 T628 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 103277 T628 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38396/solr
   [junit4]   2> 103277 T628 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 103278 T628 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 103280 T670 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cbfd44 name:ZooKeeperConnection Watcher:127.0.0.1:38396 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 103281 T628 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 103283 T628 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 103284 T672 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@71f8e7 name:ZooKeeperConnection Watcher:127.0.0.1:38396/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 103284 T628 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 103288 T628 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 103379 T672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 103379 T650 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 103379 T659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 103421 T657 oasc.ZkController.register We are http://127.0.0.1:37400/yi_oj/collection1/ and leader is http://127.0.0.1:37400/yi_oj/collection1/
   [junit4]   2> 103421 T657 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:37400/yi_oj
   [junit4]   2> 103422 T657 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 103422 T657 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 103423 T657 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 103425 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 103426 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 103426 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 103426 T657 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 103429 T651 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 103431 T651 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:37400/yi_oj",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:37400_yi_oj",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 103437 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 103541 T650 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 103541 T659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 103541 T672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 104292 T628 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56128_yi_oj
   [junit4]   2> 104294 T628 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56128_yi_oj
   [junit4]   2> 104297 T659 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 104297 T672 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 104297 T650 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 104312 T673 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 104313 T673 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 104434 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 104435 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 104435 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 104434 T673 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 104437 T651 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 104438 T651 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:56128/yi_oj",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:56128_yi_oj",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 104438 T651 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 104438 T651 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 104442 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 104545 T650 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 104545 T659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 104545 T672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 105435 T673 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 105436 T673 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1389358619282/collection1
   [junit4]   2> 105436 T673 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 105437 T673 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 105437 T673 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 105439 T673 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1389358619282/collection1/'
   [junit4]   2> 105439 T673 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1389358619282/collection1/lib/classes/' to classloader
   [junit4]   2> 105440 T673 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1389358619282/collection1/lib/README' to classloader
   [junit4]   2> 105478 T673 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 105519 T673 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 105521 T673 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 105528 T673 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 105894 T673 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 105900 T673 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 105902 T673 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 105918 T673 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 105921 T673 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 105924 T673 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 105926 T673 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 105926 T673 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 105926 T673 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 105928 T673 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 105928 T673 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 105928 T673 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 105929 T673 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1389358619282/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1389358617189/jetty1/
   [junit4]   2> 105929 T673 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5f6fc9
   [junit4]   2> 105930 T673 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1389358617189/jetty1
   [junit4]   2> 105930 T673 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1389358617189/jetty1/index/
   [junit4]   2> 105930 T673 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1389358617189/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 105931 T673 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1389358617189/jetty1/index
   [junit4]   2> 105932 T673 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1919050342, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 105933 T673 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/index5769757699tmp lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@14559bc),segFN=segments_1,generation=1}
   [junit4]   2> 105933 T673 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 105937 T673 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 105938 T673 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 105938 T673 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 105938 T673 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 105939 T673 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 105939 T673 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 105939 T673 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 105940 T673 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 105940 T673 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 105941 T673 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 105941 T673 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 105942 T673 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 105942 T673 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 105943 T673 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 105943 T673 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 105944 T673 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 105945 T673 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 105947 T673 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 105949 T673 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 105949 T673 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 105950 T673 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=956245469, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 105951 T673 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/index5769757699tmp lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@14559bc),segFN=segments_1,generation=1}
   [junit4]   2> 105951 T673 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 105951 T673 oass.SolrIndexSearcher.<init> Opening Searcher@193583c main
   [junit4]   2> 105954 T674 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@193583c main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 105956 T673 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 105957 T628 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 105957 T677 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:56128/yi_oj collection:collection1 shard:shard2
   [junit4]   2> 105958 T628 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 105959 T677 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 105967 T677 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 105970 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 105970 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 105970 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 105971 T677 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 105971 T677 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C234 name=collection1 org.apache.solr.core.SolrCore@65856 url=http://127.0.0.1:56128/yi_oj/collection1 node=127.0.0.1:56128_yi_oj C234_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:56128/yi_oj, core=collection1, node_name=127.0.0.1:56128_yi_oj}
   [junit4]   2> 105971 T677 C234 P56128 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:56128/yi_oj/collection1/
   [junit4]   2> 105972 T677 C234 P56128 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 105972 T677 C234 P56128 oasc.SyncStrategy.syncToMe http://127.0.0.1:56128/yi_oj/collection1/ has no replicas
   [junit4]   2> 105972 T651 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 105972 T677 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:56128/yi_oj/collection1/ shard2
   [junit4]   2> 105973 T677 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 105977 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 105980 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 106042 T628 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 106044 T628 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55435
   [junit4]   2> 106045 T628 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 106045 T628 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 106045 T628 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1389358622115
   [junit4]   2> 106046 T628 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1389358622115/'
   [junit4]   2> 106055 T628 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1389358622115/solr.xml
   [junit4]   2> 106087 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 106089 T659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 106089 T672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 106089 T650 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 106093 T628 oasc.CoreContainer.<init> New CoreContainer 582552
   [junit4]   2> 106094 T628 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1389358622115/]
   [junit4]   2> 106095 T628 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 106095 T628 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 106095 T628 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 106096 T628 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 106096 T628 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 106096 T628 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 106097 T628 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 106097 T628 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 106098 T628 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 106101 T628 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 106101 T628 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 106102 T628 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 106102 T628 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38396/solr
   [junit4]   2> 106102 T628 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 106103 T628 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 106105 T688 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d67edd name:ZooKeeperConnection Watcher:127.0.0.1:38396 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 106106 T628 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 106109 T628 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 106111 T690 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ae3d name:ZooKeeperConnection Watcher:127.0.0.1:38396/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 106111 T628 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 106119 T628 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 106137 T677 oasc.ZkController.register We are http://127.0.0.1:56128/yi_oj/collection1/ and leader is http://127.0.0.1:56128/yi_oj/collection1/
   [junit4]   2> 106138 T677 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:56128/yi_oj
   [junit4]   2> 106138 T677 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 106139 T677 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 106139 T677 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 106141 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 106142 T677 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 106144 T651 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 106145 T651 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:56128/yi_oj",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:56128_yi_oj",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 106148 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 106252 T690 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 106253 T672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 106252 T659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 106252 T650 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 107123 T628 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55435_yi_oj
   [junit4]   2> 107125 T628 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55435_yi_oj
   [junit4]   2> 107127 T672 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 107127 T650 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 107127 T659 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 107127 T690 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 107202 T691 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 107202 T691 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 107204 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 107204 T691 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 107205 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 107205 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 107208 T651 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 107210 T651 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:55435/yi_oj",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:55435_yi_oj",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 107210 T651 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 107210 T651 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 107215 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 107318 T650 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 107318 T690 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 107318 T672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 107318 T659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 108205 T691 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 108205 T691 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1389358622115/collection1
   [junit4]   2> 108205 T691 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 108206 T691 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 108207 T691 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 108208 T691 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1389358622115/collection1/'
   [junit4]   2> 108208 T691 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1389358622115/collection1/lib/classes/' to classloader
   [junit4]   2> 108209 T691 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1389358622115/collection1/lib/README' to classloader
   [junit4]   2> 108243 T691 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 108281 T691 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 108283 T691 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 108292 T691 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 108623 T691 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 108628 T691 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 108631 T691 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 108647 T691 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 108650 T691 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 108653 T691 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 108655 T691 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 108655 T691 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 108655 T691 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 108656 T691 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 108657 T691 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 108657 T691 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 108657 T691 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1389358622115/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1389358617189/jetty2/
   [junit4]   2> 108657 T691 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5f6fc9
   [junit4]   2> 108658 T691 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1389358617189/jetty2
   [junit4]   2> 108658 T691 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1389358617189/jetty2/index/
   [junit4]   2> 108658 T691 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1389358617189/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 108659 T691 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1389358617189/jetty2/index
   [junit4]   2> 108659 T691 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1919050342, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 108660 T691 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/index5769757700tmp lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1e992a),segFN=segments_1,generation=1}
   [junit4]   2> 108661 T691 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 108664 T691 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 108664 T691 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 108664 T691 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 108665 T691 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 108665 T691 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 108665 T691 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 108665 T691 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 108666 T691 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 108666 T691 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 108666 T691 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 108667 T691 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 108667 T691 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 108667 T691 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 108668 T691 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 108668 T691 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 108669 T691 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 108669 T691 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 108671 T691 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 108673 T691 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 108673 T691 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 108674 T691 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=956245469, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 108675 T691 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/index5769757700tmp lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1e992a),segFN=segments_1,generation=1}
   [junit4]   2> 108675 T691 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 108676 T691 oass.SolrIndexSearcher.<init> Opening Searcher@1122a34 main
   [junit4]   2> 108679 T692 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1122a34 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 108681 T691 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 108682 T628 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 108682 T628 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 108682 T695 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:55435/yi_oj collection:collection1 shard:shard1
   [junit4]   2> 108684 T695 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 108691 T695 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 108693 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 108693 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 108693 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 108694 T695 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 108694 T695 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C235 name=collection1 org.apache.solr.core.SolrCore@1162ec8 url=http://127.0.0.1:55435/yi_oj/collection1 node=127.0.0.1:55435_yi_oj C235_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:55435/yi_oj, core=collection1, node_name=127.0.0.1:55435_yi_oj}
   [junit4]   2> 108694 T695 C235 P55435 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:55435/yi_oj/collection1/
   [junit4]   2> 108694 T695 C235 P55435 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 108695 T695 C235 P55435 oasc.SyncStrategy.syncToMe http://127.0.0.1:55435/yi_oj/collection1/ has no replicas
   [junit4]   2> 108695 T695 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:55435/yi_oj/collection1/ shard1
   [junit4]   2> 108695 T651 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 108695 T695 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 108698 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 108700 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 108720 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 108758 T628 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 108760 T628 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57075
   [junit4]   2> 108761 T628 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 108761 T628 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 108762 T628 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1389358624839
   [junit4]   2> 108762 T628 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1389358624839/'
   [junit4]   2> 108771 T628 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1389358624839/solr.xml
   [junit4]   2> 108808 T628 oasc.CoreContainer.<init> New CoreContainer 5813024
   [junit4]   2> 108809 T628 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1389358624839/]
   [junit4]   2> 108810 T628 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 108810 T628 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 108810 T628 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 108810 T628 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 108811 T628 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 108811 T628 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 108811 T628 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 108812 T628 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 108812 T628 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 108815 T628 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 108815 T628 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 108815 T628 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 108816 T628 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38396/solr
   [junit4]   2> 108816 T628 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 108817 T628 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 108819 T706 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1580b7d name:ZooKeeperConnection Watcher:127.0.0.1:38396 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 108819 T628 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 108822 T628 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 108823 T650 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 108823 T690 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 108823 T672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 108823 T659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 108824 T708 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@44a342 name:ZooKeeperConnection Watcher:127.0.0.1:38396/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 108825 T628 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 108835 T628 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 108852 T695 oasc.ZkController.register We are http://127.0.0.1:55435/yi_oj/collection1/ and leader is http://127.0.0.1:55435/yi_oj/collection1/
   [junit4]   2> 108852 T695 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:55435/yi_oj
   [junit4]   2> 108852 T695 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 108852 T695 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 108852 T695 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 108854 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 108854 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 108854 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 108855 T695 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 108856 T651 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 108857 T651 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:55435/yi_oj",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:55435_yi_oj",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 108863 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 108984 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 108984 T690 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 108984 T659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 108984 T672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 108985 T650 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 109839 T628 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57075_yi_oj
   [junit4]   2> 109841 T628 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57075_yi_oj
   [junit4]   2> 109845 T650 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 109845 T672 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 109846 T659 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 109846 T690 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 109846 T708 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 109860 T709 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 109860 T709 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 109862 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 109862 T709 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 109863 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 109863 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 109866 T651 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 109868 T651 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57075/yi_oj",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57075_yi_oj",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 109868 T651 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 109868 T651 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 109874 T650 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 109979 T672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 109979 T659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 109979 T690 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 109979 T650 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 109979 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 110863 T709 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 110864 T709 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1389358624839/collection1
   [junit4]   2> 110864 T709 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 110865 T709 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 110865 T709 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 110866 T709 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1389358624839/collection1/'
   [junit4]   2> 110867 T709 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1389358624839/collection1/lib/classes/' to classloader
   [junit4]   2> 110867 T709 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1389358624839/collection1/lib/README' to classloader
   [junit4]   2> 110913 T709 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 110956 T709 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 110959 T709 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 110967 T709 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 111336 T709 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 111342 T709 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 111344 T709 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 111367 T709 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 111371 T709 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 111374 T709 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 111376 T709 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 111376 T709 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 111376 T709 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 111377 T709 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 111378 T709 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 111378 T709 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 111378 T709 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1389358624839/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1389358617189/jetty3/
   [junit4]   2> 111378 T709 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5f6fc9
   [junit4]   2> 111379 T709 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1389358617189/jetty3
   [junit4]   2> 111379 T709 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1389358617189/jetty3/index/
   [junit4]   2> 111379 T709 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1389358617189/jetty3/index' doesn't exist. Creating new index...
   [junit4]   2> 111380 T709 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1389358617189/jetty3/index
   [junit4]   2> 111380 T709 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1919050342, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 111382 T709 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/index5769757701tmp lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@90b9ba),segFN=segments_1,generation=1}
   [junit4]   2> 111382 T709 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 111385 T709 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 111386 T709 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 111386 T709 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 111386 T709 oasup.UpdateRequestProcessorChain.init creating upda

[...truncated too long message...]

ild/solr-core/test/J0/./solrtest-BasicDistributedZkTest-1389358617190/collection24
   [junit4]   2> 206962 T727 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection3/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:91)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:136)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 206963 T727 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 206963 T727 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:91)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:136)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 206964 T727 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 206964 T727 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 206964 T727 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:91)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:136)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 105968 T627 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> 126046 T627 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest: 
   [junit4]   2> 	   1) Thread[id=1061, name=Timer-0, state=WAITING, group=TGRP-BasicDistributedZkTest]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.lang.Object.wait(Object.java:503)
   [junit4]   2> 	        at java.util.TimerThread.mainLoop(Timer.java:526)
   [junit4]   2> 	        at java.util.TimerThread.run(Timer.java:505)
   [junit4]   2> 126046 T627 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
   [junit4]   2> 	   1) Thread[id=1061, name=Timer-0, state=WAITING, group=TGRP-BasicDistributedZkTest]
   [junit4]   2> 129049 T627 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
   [junit4]   2> 	   1) Thread[id=1061, name=Timer-0, state=WAITING, group=TGRP-BasicDistributedZkTest]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.lang.Object.wait(Object.java:503)
   [junit4]   2> 	        at java.util.TimerThread.mainLoop(Timer.java:526)
   [junit4]   2> 	        at java.util.TimerThread.run(Timer.java:505)
   [junit4]   2> NOTE: test params are: codec=Lucene41, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=en_NZ, timezone=America/Halifax
   [junit4]   2> NOTE: Linux 3.8.0-35-generic i386/Oracle Corporation 1.7.0_45 (32-bit)/cpus=8,threads=2,free=46632488,total=134115328
   [junit4]   2> NOTE: All tests run in this JVM: [CSVRequestHandlerTest, TestCustomSort, OverseerCollectionProcessorTest, IndexSchemaTest, TestSolrXml, TestReloadAndDeleteDocs, TestSolrIndexConfig, PeerSyncTest, RecoveryZkTest, TestSystemIdResolver, TestWriterPerf, AbstractAnalyticsFacetTest, CacheHeaderTest, TestElisionMultitermQuery, TestSurroundQueryParser, TestLuceneMatchVersion, UpdateRequestProcessorFactoryTest, SynonymTokenizerTest, InfoHandlerTest, LoggingHandlerTest, DistributedTermsComponentTest, SolrCoreTest, UUIDFieldTest, CoreAdminCreateDiscoverTest, SimpleFacetsTest, DistributedSpellCheckComponentTest, FastVectorHighlighterTest, TestDocSet, HdfsSyncSliceTest, DateMathParserTest, ClusterStateTest, SearchHandlerTest, BasicDistributedZkTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=BasicDistributedZkTest -Dtests.seed=113AA8383E83B11D -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en_NZ -Dtests.timezone=America/Halifax -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J0 | BasicDistributedZkTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest: 
   [junit4]    >    1) Thread[id=1061, name=Timer-0, state=WAITING, group=TGRP-BasicDistributedZkTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at java.lang.Object.wait(Object.java:503)
   [junit4]    >         at java.util.TimerThread.mainLoop(Timer.java:526)
   [junit4]    >         at java.util.TimerThread.run(Timer.java:505)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([113AA8383E83B11D]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=1061, name=Timer-0, state=WAITING, group=TGRP-BasicDistributedZkTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at java.lang.Object.wait(Object.java:503)
   [junit4]    >         at java.util.TimerThread.mainLoop(Timer.java:526)
   [junit4]    >         at java.util.TimerThread.run(Timer.java:505)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([113AA8383E83B11D]:0)
   [junit4] Completed on J0 in 129.09s, 1 test, 2 errors <<< FAILURES!

[...truncated 984 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:453: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:433: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1304: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:937: There were test failures: 362 suites, 1589 tests, 2 suite-level errors, 1047 ignored (4 assumptions)

Total time: 35 minutes 29 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_45 -server -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure