You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/02/18 10:30:56 UTC

[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.7.0_60-ea-b04) - Build # 9407 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9407/
Java: 64bit/jdk1.7.0_60-ea-b04 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([8D2083A9F9731DF1]:0)


REGRESSION:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([8D2083A9F9731DF1]:0)




Build Log:
[...truncated 11017 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> 15092 T59 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 15092 T59 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 15100 T59 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeySafeLeaderTest-1392708603291
   [junit4]   2> 15103 T59 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 15105 T60 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 15205 T59 oasc.ZkTestServer.run start zk server on port:38204
   [junit4]   2> 15255 T59 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 15275 T61 oazs.NIOServerCnxn.doIO WARN Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
   [junit4]   2> 16876 T66 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2fd36711 name:ZooKeeperConnection Watcher:127.0.0.1:38204 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 16877 T59 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 16879 T59 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 16909 T59 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 16911 T68 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3118a325 name:ZooKeeperConnection Watcher:127.0.0.1:38204/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 16911 T59 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 16916 T59 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 16921 T59 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 16927 T59 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 16931 T59 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 16936 T59 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 16936 T59 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 16943 T59 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 16944 T59 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 16949 T59 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 16949 T59 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 16954 T59 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 16955 T59 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 16960 T59 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 16960 T59 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 16965 T59 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 16965 T59 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 16969 T59 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 16969 T59 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 16973 T59 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 16973 T59 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 16977 T59 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 16977 T59 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 16981 T59 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 16981 T59 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 16986 T59 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 16986 T59 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 17177 T59 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 17250 T59 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:35581
   [junit4]   2> 17277 T59 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 17278 T59 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 17278 T59 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392708605183
   [junit4]   2> 17278 T59 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392708605183/'
   [junit4]   2> 17289 T59 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392708605183/solr.xml
   [junit4]   2> 17319 T59 oasc.CoreContainer.<init> New CoreContainer 1955836476
   [junit4]   2> 17320 T59 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392708605183/]
   [junit4]   2> 17329 T59 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 17329 T59 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 17330 T59 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 17330 T59 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 17330 T59 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 17331 T59 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 17331 T59 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 17331 T59 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 17332 T59 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 17335 T59 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 17335 T59 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 17335 T59 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 17336 T59 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38204/solr
   [junit4]   2> 17343 T59 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 17344 T59 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 17348 T80 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3d6271bb name:ZooKeeperConnection Watcher:127.0.0.1:38204 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 17349 T59 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 17356 T59 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 17360 T82 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2cb88b36 name:ZooKeeperConnection Watcher:127.0.0.1:38204/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 17360 T59 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 17368 T59 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 17382 T59 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 17396 T59 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 17402 T59 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35581_
   [junit4]   2> 17407 T59 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35581_
   [junit4]   2> 17418 T59 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 17425 T59 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 17446 T59 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:35581_
   [junit4]   2> 17447 T59 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 17454 T59 oasc.Overseer.start Overseer (id=91272551030849539-127.0.0.1:35581_-n_0000000000) starting
   [junit4]   2> 17468 T59 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 17483 T84 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 17484 T59 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 17491 T59 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 17493 T84 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 17498 T59 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 17513 T83 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 17522 T85 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 17523 T85 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 17530 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 17530 T85 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 17536 T83 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 17539 T83 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35581",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35581_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 17540 T83 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 17553 T83 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 17561 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 17569 T82 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 18531 T85 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 18532 T85 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392708605183/collection1
   [junit4]   2> 18532 T85 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 18533 T85 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 18534 T85 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 18535 T85 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 18536 T85 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392708605183/collection1/'
   [junit4]   2> 18537 T85 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392708605183/collection1/lib/classes/' to classloader
   [junit4]   2> 18537 T85 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392708605183/collection1/lib/README' to classloader
   [junit4]   2> 18594 T85 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 18622 T85 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 18624 T85 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 18636 T85 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 18895 T85 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 18898 T85 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 18900 T85 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 18906 T85 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 18957 T85 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 18963 T85 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392708605183/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/control/data/
   [junit4]   2> 18963 T85 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@f3f6de1
   [junit4]   2> 18967 T85 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/control/data
   [junit4]   2> 18967 T85 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/control/data/index/
   [junit4]   2> 18967 T85 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 18968 T85 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/control/data/index
   [junit4]   2> 18968 T85 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=34, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8205642743276823]
   [junit4]   2> 18971 T85 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 18972 T85 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 18979 T85 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 18980 T85 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 18981 T85 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 18981 T85 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 18981 T85 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 18982 T85 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 18982 T85 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 18983 T85 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 18984 T85 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 18986 T85 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 18987 T85 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 18987 T85 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 18988 T85 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 18988 T85 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 18989 T85 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 18990 T85 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 18996 T85 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 18998 T85 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 18998 T85 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 19000 T85 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=27, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.26835472916702285]
   [junit4]   2> 19002 T85 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 19002 T85 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 19002 T85 oass.SolrIndexSearcher.<init> Opening Searcher@7c08f261[collection1] main
   [junit4]   2> 19010 T86 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7c08f261[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 19015 T85 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 19016 T59 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 19016 T89 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:35581 collection:control_collection shard:shard1
   [junit4]   2> 19016 T59 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 19023 T89 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 19034 T59 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 19036 T89 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 19038 T91 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2247c9cd name:ZooKeeperConnection Watcher:127.0.0.1:38204/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 19038 T59 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 19041 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 19041 T59 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 19042 T89 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 19043 T89 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 19044 T83 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C8 name=collection1 org.apache.solr.core.SolrCore@64de9461 url=http://127.0.0.1:35581/collection1 node=127.0.0.1:35581_ C8_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:35581, core=collection1, node_name=127.0.0.1:35581_}
   [junit4]   2> 19044 T89 C8 P35581 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:35581/collection1/
   [junit4]   2> 19045 T89 C8 P35581 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 19045 T89 C8 P35581 oasc.SyncStrategy.syncToMe http://127.0.0.1:35581/collection1/ has no replicas
   [junit4]   2> 19046 T89 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:35581/collection1/ shard1
   [junit4]   2> 19047 T89 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 19048 T59 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 19054 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 19062 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 19067 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 19069 T82 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 19069 T91 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 19112 T89 oasc.ZkController.register We are http://127.0.0.1:35581/collection1/ and leader is http://127.0.0.1:35581/collection1/
   [junit4]   2> 19113 T89 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:35581
   [junit4]   2> 19113 T89 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 19113 T89 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 19114 T89 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 19115 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 19116 T89 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 19118 T83 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 19119 T83 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35581",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35581_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 19124 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 19151 T59 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 19152 T59 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 19154 T59 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52491
   [junit4]   2> 19155 T59 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 19156 T59 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 19156 T59 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392708607239
   [junit4]   2> 19157 T59 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392708607239/'
   [junit4]   2> 19169 T59 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392708607239/solr.xml
   [junit4]   2> 19198 T59 oasc.CoreContainer.<init> New CoreContainer 135178557
   [junit4]   2> 19198 T59 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392708607239/]
   [junit4]   2> 19199 T59 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 19199 T59 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 19199 T59 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 19200 T59 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 19200 T59 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 19200 T59 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 19201 T59 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 19201 T59 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 19201 T59 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 19204 T59 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 19204 T59 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 19205 T59 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 19205 T59 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38204/solr
   [junit4]   2> 19205 T59 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 19206 T59 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 19209 T102 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a88e7ec name:ZooKeeperConnection Watcher:127.0.0.1:38204 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 19210 T59 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 19215 T59 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 19218 T104 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5443de84 name:ZooKeeperConnection Watcher:127.0.0.1:38204/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 19218 T59 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 19228 T82 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 19228 T91 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 19232 T59 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 20241 T59 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52491_
   [junit4]   2> 20243 T59 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52491_
   [junit4]   2> 20247 T91 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 20247 T104 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 20247 T82 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 20256 T105 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 20256 T105 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 20258 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 20258 T105 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 20258 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 20259 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 20261 T83 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 20262 T83 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52491",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52491_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 20262 T83 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 20262 T83 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 20267 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 20371 T82 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 20371 T91 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 20371 T104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 21258 T105 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 21259 T105 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392708607239/collection1
   [junit4]   2> 21259 T105 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 21260 T105 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 21260 T105 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 21262 T105 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 21262 T105 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392708607239/collection1/'
   [junit4]   2> 21263 T105 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392708607239/collection1/lib/classes/' to classloader
   [junit4]   2> 21263 T105 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392708607239/collection1/lib/README' to classloader
   [junit4]   2> 21309 T105 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 21335 T105 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 21338 T105 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 21347 T105 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 21577 T105 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 21579 T105 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 21581 T105 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 21585 T105 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 21633 T105 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 21633 T105 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392708607239/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty1/
   [junit4]   2> 21634 T105 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@f3f6de1
   [junit4]   2> 21636 T105 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty1
   [junit4]   2> 21636 T105 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty1/index/
   [junit4]   2> 21636 T105 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 21637 T105 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty1/index
   [junit4]   2> 21637 T105 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=34, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8205642743276823]
   [junit4]   2> 21639 T105 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 21640 T105 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 21650 T105 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 21651 T105 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 21651 T105 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 21652 T105 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 21652 T105 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 21652 T105 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 21653 T105 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 21654 T105 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 21654 T105 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 21655 T105 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 21657 T105 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 21657 T105 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 21658 T105 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 21660 T105 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 21660 T105 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 21661 T105 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 21663 T105 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 21665 T105 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 21665 T105 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 21667 T105 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=27, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.26835472916702285]
   [junit4]   2> 21668 T105 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 21668 T105 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 21669 T105 oass.SolrIndexSearcher.<init> Opening Searcher@33157362[collection1] main
   [junit4]   2> 21682 T106 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@33157362[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 21691 T105 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 21692 T59 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 21693 T59 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 21692 T109 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52491 collection:collection1 shard:shard2
   [junit4]   2> 21696 T109 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 21710 T109 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 21715 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 21715 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 21716 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 21716 T109 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 21716 T109 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C9 name=collection1 org.apache.solr.core.SolrCore@1eb42164 url=http://127.0.0.1:52491/collection1 node=127.0.0.1:52491_ C9_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:52491, core=collection1, node_name=127.0.0.1:52491_}
   [junit4]   2> 21717 T109 C9 P52491 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52491/collection1/
   [junit4]   2> 21718 T109 C9 P52491 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 21718 T109 C9 P52491 oasc.SyncStrategy.syncToMe http://127.0.0.1:52491/collection1/ has no replicas
   [junit4]   2> 21719 T109 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52491/collection1/ shard2
   [junit4]   2> 21719 T83 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 21719 T109 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 21725 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 21728 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 21733 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 21782 T59 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 21783 T59 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 21785 T59 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:48646
   [junit4]   2> 21786 T59 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 21786 T59 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 21787 T59 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392708609884
   [junit4]   2> 21787 T59 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392708609884/'
   [junit4]   2> 21806 T59 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392708609884/solr.xml
   [junit4]   2> 21835 T59 oasc.CoreContainer.<init> New CoreContainer 458465728
   [junit4]   2> 21836 T59 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392708609884/]
   [junit4]   2> 21837 T59 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 21838 T104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 21838 T82 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 21838 T91 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 21838 T59 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 21839 T59 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 21840 T59 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 21840 T59 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 21841 T59 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 21841 T59 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 21841 T59 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 21842 T59 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 21845 T59 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 21846 T59 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 21846 T59 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 21846 T59 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38204/solr
   [junit4]   2> 21847 T59 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 21848 T59 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 21849 T120 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@50f752ee name:ZooKeeperConnection Watcher:127.0.0.1:38204 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 21850 T59 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 21854 T59 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 21856 T122 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@39978617 name:ZooKeeperConnection Watcher:127.0.0.1:38204/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 21857 T59 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 21868 T59 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 21880 T109 oasc.ZkController.register We are http://127.0.0.1:52491/collection1/ and leader is http://127.0.0.1:52491/collection1/
   [junit4]   2> 21880 T109 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52491
   [junit4]   2> 21880 T109 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 21881 T109 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 21881 T109 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 21884 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 21884 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 21885 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 21885 T109 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 21889 T83 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 21892 T83 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52491",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52491_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 21899 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 22005 T91 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 22005 T104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 22005 T122 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 22005 T82 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 22877 T59 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48646_
   [junit4]   2> 22880 T59 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48646_
   [junit4]   2> 22887 T104 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 22887 T82 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 22888 T122 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 22887 T91 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 22903 T123 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 22903 T123 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 22906 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 22906 T123 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 22906 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 22907 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 22912 T83 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 22914 T83 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48646",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:48646_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 22915 T83 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 22915 T83 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 22923 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 23029 T82 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 23029 T91 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 23030 T122 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 23029 T104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 23907 T123 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 23908 T123 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392708609884/collection1
   [junit4]   2> 23908 T123 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 23917 T123 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 23917 T123 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 23919 T123 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 23919 T123 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392708609884/collection1/'
   [junit4]   2> 23920 T123 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392708609884/collection1/lib/classes/' to classloader
   [junit4]   2> 23921 T123 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392708609884/collection1/lib/README' to classloader
   [junit4]   2> 23962 T123 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 23991 T123 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 23993 T123 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 24001 T123 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 24254 T123 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 24255 T123 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 24257 T123 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 24260 T123 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 24305 T123 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 24306 T123 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392708609884/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty2/
   [junit4]   2> 24306 T123 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@f3f6de1
   [junit4]   2> 24307 T123 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty2
   [junit4]   2> 24308 T123 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty2/index/
   [junit4]   2> 24308 T123 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 24309 T123 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty2/index
   [junit4]   2> 24310 T123 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=34, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8205642743276823]
   [junit4]   2> 24400 T123 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 24400 T123 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 24408 T123 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 24408 T123 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 24409 T123 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 24409 T123 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 24409 T123 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 24409 T123 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 24410 T123 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 24410 T123 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 24410 T123 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 24412 T123 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 24413 T123 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 24414 T123 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 24414 T123 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 24415 T123 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 24416 T123 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 24417 T123 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 24419 T123 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 24421 T123 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 24421 T123 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 24423 T123 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=27, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.26835472916702285]
   [junit4]   2> 24425 T123 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 24425 T123 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 24425 T123 oass.SolrIndexSearcher.<init> Opening Searcher@109a311d[collection1] main
   [junit4]   2> 24436 T124 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@109a311d[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 24443 T123 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 24444 T59 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 24445 T59 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 24444 T127 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:48646 collection:collection1 shard:shard1
   [junit4]   2> 24452 T127 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 24468 T127 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 24472 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 24472 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 24472 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 24473 T127 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 24473 T127 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C10 name=collection1 org.apache.solr.core.SolrCore@5512520e url=http://127.0.0.1:48646/collection1 node=127.0.0.1:48646_ C10_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:48646, core=collection1, node_name=127.0.0.1:48646_}
   [junit4]   2> 24473 T127 C10 P48646 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:48646/collection1/
   [junit4]   2> 24473 T127 C10 P48646 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 24474 T127 C10 P48646 oasc.SyncStrategy.syncToMe http://127.0.0.1:48646/collection1/ has no replicas
   [junit4]   2> 24474 T127 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:48646/collection1/ shard1
   [junit4]   2> 24474 T127 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 24477 T83 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 24487 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 24490 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 24543 T59 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 24544 T59 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 24546 T59 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:43362
   [junit4]   2> 24547 T59 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 24547 T59 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 24547 T59 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392708612636
   [junit4]   2> 24548 T59 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392708612636/'
   [junit4]   2> 24568 T59 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392708612636/solr.xml
   [junit4]   2> 24601 T82 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 24601 T91 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 24601 T104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 24603 T122 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 24618 T59 oasc.CoreContainer.<init> New CoreContainer 1165359000
   [junit4]   2> 24619 T59 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392708612636/]
   [junit4]   2> 24620 T59 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 24621 T59 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 24621 T59 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 24622 T59 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 24622 T59 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 24623 T59 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 24623 T59 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 24623 T59 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 24624 T59 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 24628 T59 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 24629 T59 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 24629 T59 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 24630 T59 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38204/solr
   [junit4]   2> 24630 T59 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 24631 T59 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 24634 T138 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@43d23fc8 name:ZooKeeperConnection Watcher:127.0.0.1:38204 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 24635 T59 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 24637 T127 oasc.ZkController.register We are http://127.0.0.1:48646/collection1/ and leader is http://127.0.0.1:48646/collection1/
   [junit4]   2> 24638 T61 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14443e6cdf00009, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 24638 T59 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 24638 T127 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:48646
   [junit4]   2> 24639 T127 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 24639 T127 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 24640 T127 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 24640 T140 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a1871df name:ZooKeeperConnection Watcher:127.0.0.1:38204/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 24641 T59 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 24641 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 24642 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 24642 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 24642 T127 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 24645 T83 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 24646 T83 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48646",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:48646_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 24648 T59 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 24650 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 24753 T82 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 24753 T122 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 24753 T91 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 24753 T104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 24753 T140 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 25653 T59 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43362_
   [junit4]   2> 25655 T59 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43362_
   [junit4]   2> 25658 T122 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 25658 T91 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 25659 T104 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 25659 T140 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 25659 T82 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 25667 T141 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 25668 T141 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 25670 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 25670 T141 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 25670 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 25671 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 25673 T83 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 25674 T83 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:43362",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:43362_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 25675 T83 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 25675 T83 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 25681 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 25784 T82 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 25785 T140 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 25785 T91 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 25785 T122 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 25784 T104 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 26671 T141 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 26671 T141 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392708612636/collection1
   [junit4]   2> 26672 T141 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 26674 T141 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 26674 T141 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 26677 T141 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 26677 T141 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392708612636/collection1/'
   [junit4]   2> 26679 T141 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392708612636/collection1/lib/classes/' to classloader
   [junit4]   2> 26680 T141 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392708612636/collection1/lib/README' to classloader
   [junit4]   2> 26720 T141 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 26746 T141 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 26750 T141 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 26759 T141 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 26996 T141 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 26998 T141 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 26999 T141 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 27003 T141 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 27041 T141 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 27041 T141 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392708612636/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty3/
   [junit4]   2> 27042 T141 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@f3f6de1
   [junit4]   2> 27043 T141 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty3
   [junit4]   2> 27043 T141 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty3/index/
   [junit4]   2> 27043 T141 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty3/index' doesn't exist. Creating new index...
   [junit4]   2> 27044 T141 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty3/index
   [junit4]   2> 27044 T141 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=34, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8205642743276823]
   [junit4]   2> 27046 T141 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty3/index,segFN=segments_1,generation=1}
   [junit4]   2> 27047 T141 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 27054 T141 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 27055 T141 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 27055 T141 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 27055 T141 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 27056 T141 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 27056 T141 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 27056 T141 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 27057 T141 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 27057 T141 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 27058 T141 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 27059 T141 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 27059 T141 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 27060 T141 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 27061 T141 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 27061 T141 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 27062 T141 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 27064 T141 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 27066 T141 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 27067 T141 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 27068 T141 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=27, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.26835472916702285]
   [junit4]   2> 27069 T141 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392708603291/jetty3/index,segFN=segments_1,generation=1}
   [junit4]   2> 27070 T141 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 27070 T141 oass.SolrIndexSearcher.<init> Opening Searcher@69eb0ef1[collection1] main
   [junit4]   2> 27077 T142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@69eb0ef1[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 27082 T141 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 27082 T59 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 27083 T59 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 27082 T145 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:43362 collection:collection1 shard:shard3
   [junit4]   2> 27085 T145 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 27095 T145 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 27098 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27098 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27099 T145 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 27099 T82 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27099 T145 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C11 name=collection1 org.apache.solr.core.SolrCore@19080dcc url=http://127.0.0.1:43362/collection1 node=127.0.0.1:43362_ C11_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:43362, core=collection1, node_name=127.0.0.1:43362_}
   [junit4]   2> 27100 T145 C11 P43362 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:43362/collection1/
   [junit4]   2> 27100 T145 C11 P43362 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 27100 T1

[...truncated too long message...]

   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	  68) Thread[id=235, name=TEST-ChaosMonkeySafeLeaderTest.testDistribSearch-seed#[8D2083A9F9731DF1]-SendThread(localhost.localdomain:38204), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 	  69) Thread[id=286, name=qtp2002619729-286 Selector1, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
   [junit4]   2> 	        at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)
   [junit4]   2> 	        at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:193)
   [junit4]   2> 	        at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:210)
   [junit4]   2> 	        at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:216)
   [junit4]   2> 	        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:78)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:106)
   [junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:536)
   [junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	  70) Thread[id=227, name=qtp485339482-227 Acceptor0 SelectChannelConnector@127.0.0.1:52063, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:956)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	  71) Thread[id=535, name=coreZkRegister-237-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	  72) Thread[id=288, name=qtp2002619729-288 Acceptor1 SelectChannelConnector@127.0.0.1:55023, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:956)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	  73) Thread[id=547, name=qtp859089295-547 Selector1, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
   [junit4]   2> 	        at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)
   [junit4]   2> 	        at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:193)
   [junit4]   2> 	        at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:210)
   [junit4]   2> 	        at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:216)
   [junit4]   2> 	        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:78)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
   [junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
   [junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	  74) Thread[id=342, name=commitScheduler-134-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2> 	        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]   2> 	        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	  75) Thread[id=284, name=HashSessionScavenger-11, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.util.TimerThread.mainLoop(Timer.java:552)
   [junit4]   2> 	        at java.util.TimerThread.run(Timer.java:505)
   [junit4]   2> NOTE: test params are: codec=Lucene46: {id=PostingsFormat(name=Direct), rnd_b=PostingsFormat(name=Direct), _version_=PostingsFormat(name=Asserting), a_t=PostingsFormat(name=Asserting), a_si=PostingsFormat(name=TestBloomFilteredLucene41Postings)}, docValues:{}, sim=DefaultSimilarity, locale=it, timezone=Africa/Brazzaville
   [junit4]   2> NOTE: Linux 3.8.0-35-generic amd64/Oracle Corporation 1.7.0_60-ea (64-bit)/cpus=8,threads=87,free=35082112,total=126418944
   [junit4]   2> NOTE: All tests run in this JVM: [TestCollationFieldDocValues, DirectSolrSpellCheckerTest, DocValuesMultiTest, SolrPluginUtilsTest, SolrCoreCheckLockOnStartupTest, TestPartialUpdateDeduplication, ChangedSchemaMergeTest, TestQuerySenderNoQuery, CursorMarkTest, SpellingQueryConverterTest, ChaosMonkeySafeLeaderTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=8D2083A9F9731DF1 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=it -Dtests.timezone=Africa/Brazzaville -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J1 | ChaosMonkeySafeLeaderTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([8D2083A9F9731DF1]:0)
   [junit4] Completed on J1 in 7225.69s, 1 test, 2 errors <<< FAILURES!

[...truncated 251 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:459: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:439: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1308: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:941: There were test failures: 362 suites, 1563 tests, 1 suite-level error, 1 error, 304 ignored (5 assumptions)

Total time: 143 minutes 4 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_60-ea-b04 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Re: [JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_60-ea-b04) - Build # 9410 - Still Failing!

Posted by Simon Willnauer <si...@gmail.com>.
I committed a fix for this - sorry for the noise here!

On Tue, Feb 18, 2014 at 6:04 PM, Mark Miller <ma...@gmail.com> wrote:
> Fail is:
>
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:240: Some example solrconfig.xml files do not refer to the correct luceneMatchVersion: 4.8
>
> - Mark
>
> http://about.me/markrmiller
>
> On Feb 18, 2014, at 12:00 PM, Policeman Jenkins Server <je...@thetaphi.de> wrote:
>
>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9410/
>> Java: 32bit/jdk1.7.0_60-ea-b04 -client -XX:+UseParallelGC
>>
>> All tests passed
>>
>> Build Log:
>> [...truncated 28008 lines...]
>> BUILD FAILED
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:459: The following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:64: The following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:240: Some example solrconfig.xml files do not refer to the correct luceneMatchVersion: 4.8
>>
>> Total time: 58 minutes 12 seconds
>> Build step 'Invoke Ant' marked build as failure
>> Description set: Java: 32bit/jdk1.7.0_60-ea-b04 -client -XX:+UseParallelGC
>> Archiving artifacts
>> Recording test results
>> Email was triggered for: Failure
>> Sending email for trigger: Failure
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: [JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_60-ea-b04) - Build # 9410 - Still Failing!

Posted by Mark Miller <ma...@gmail.com>.
Fail is:

/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:240: Some example solrconfig.xml files do not refer to the correct luceneMatchVersion: 4.8

- Mark

http://about.me/markrmiller

On Feb 18, 2014, at 12:00 PM, Policeman Jenkins Server <je...@thetaphi.de> wrote:

> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9410/
> Java: 32bit/jdk1.7.0_60-ea-b04 -client -XX:+UseParallelGC
> 
> All tests passed
> 
> Build Log:
> [...truncated 28008 lines...]
> BUILD FAILED
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:459: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:64: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:240: Some example solrconfig.xml files do not refer to the correct luceneMatchVersion: 4.8
> 
> Total time: 58 minutes 12 seconds
> Build step 'Invoke Ant' marked build as failure
> Description set: Java: 32bit/jdk1.7.0_60-ea-b04 -client -XX:+UseParallelGC
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_60-ea-b04) - Build # 9410 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9410/
Java: 32bit/jdk1.7.0_60-ea-b04 -client -XX:+UseParallelGC

All tests passed

Build Log:
[...truncated 28008 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:459: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:64: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:240: Some example solrconfig.xml files do not refer to the correct luceneMatchVersion: 4.8

Total time: 58 minutes 12 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_60-ea-b04 -client -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-4.x-Linux (32bit/ibm-j9-jdk7) - Build # 9409 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9409/
Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}

4 tests failed.
REGRESSION:  org.apache.solr.handler.component.DistributedQueryElevationComponentTest.testDistribSearch

Error Message:
.responseHeader.params.rows!=version (unordered or missing)

Stack Trace:
junit.framework.AssertionFailedError: .responseHeader.params.rows!=version (unordered or missing)
	at __randomizedtesting.SeedInfo.seed([C64D18B3B757CA71:47AB96ABC008AA4D]:0)
	at junit.framework.Assert.fail(Assert.java:50)
	at org.apache.solr.BaseDistributedSearchTestCase.compareSolrResponses(BaseDistributedSearchTestCase.java:843)
	at org.apache.solr.BaseDistributedSearchTestCase.compareResponses(BaseDistributedSearchTestCase.java:862)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:565)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:545)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:524)
	at org.apache.solr.handler.component.DistributedQueryElevationComponentTest.doTest(DistributedQueryElevationComponentTest.java:81)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:613)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:780)


REGRESSION:  org.apache.solr.handler.component.TermVectorComponentDistributedTest.testDistribSearch

Error Message:
.termVectors.0.test_basictv!=test_offtv (unordered or missing)

Stack Trace:
junit.framework.AssertionFailedError: .termVectors.0.test_basictv!=test_offtv (unordered or missing)
	at __randomizedtesting.SeedInfo.seed([C64D18B3B757CA71:47AB96ABC008AA4D]:0)
	at junit.framework.Assert.fail(Assert.java:50)
	at org.apache.solr.BaseDistributedSearchTestCase.compareSolrResponses(BaseDistributedSearchTestCase.java:843)
	at org.apache.solr.BaseDistributedSearchTestCase.compareResponses(BaseDistributedSearchTestCase.java:862)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:565)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:545)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:524)
	at org.apache.solr.handler.component.TermVectorComponentDistributedTest.doTest(TermVectorComponentDistributedTest.java:137)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:876)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:613)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:780)


REGRESSION:  org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.testMultiThreaded

Error Message:
expected:<1> but was:<2>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<2>
	at __randomizedtesting.SeedInfo.seed([C64D18B3B757CA71:A7D8C2535F86C087]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.checkNumDocs(SignatureUpdateProcessorFactoryTest.java:71)
	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.testMultiThreaded(SignatureUpdateProcessorFactoryTest.java:222)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:613)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:780)


REGRESSION:  org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.testDupeDetection

Error Message:
expected:<1> but was:<3>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<3>
	at __randomizedtesting.SeedInfo.seed([C64D18B3B757CA71:9D7DE37BB410690D]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.checkNumDocs(SignatureUpdateProcessorFactoryTest.java:71)
	at org.apache.solr.update.processor.SignatureUpdateProcessorFactoryTest.testDupeDetection(SignatureUpdateProcessorFactoryTest.java:119)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:613)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:780)




Build Log:
[...truncated 10647 lines...]
   [junit4] Suite: org.apache.solr.handler.component.TermVectorComponentDistributedTest
   [junit4]   2> 1228882 T3720 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 1228883 T3720 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1228885 T3720 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 1228887 T3720 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1228888 T3720 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49891
   [junit4]   2> 1228889 T3720 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1228889 T3720 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1228890 T3720 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr
   [junit4]   2> 1228890 T3720 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/'
   [junit4]   2> 1228899 T3720 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/solr.xml
   [junit4]   2> 1228932 T3720 oasc.CoreContainer.<init> New CoreContainer -1567636093
   [junit4]   2> 1228933 T3720 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/]
   [junit4]   2> 1228934 T3720 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1228935 T3720 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1228935 T3720 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1228936 T3720 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1228937 T3720 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1228937 T3720 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1228938 T3720 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1228938 T3720 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1228939 T3720 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1228942 T3720 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1228942 T3720 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1228943 T3720 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1228946 T3730 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1
   [junit4]   2> 1228947 T3730 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/'
   [junit4]   2> 1228947 T3730 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 1228947 T3730 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 1228984 T3730 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 1229034 T3730 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1229035 T3730 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1229041 T3730 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1229376 T3730 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1229377 T3730 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1229378 T3730 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1229382 T3730 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1229385 T3730 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1229390 T3730 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1229393 T3730 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1229396 T3730 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1229397 T3730 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1229397 T3730 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1229398 T3730 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1229398 T3730 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1229398 T3730 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1229399 T3730 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1229399 T3730 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/control/data/
   [junit4]   2> 1229400 T3730 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@48da44ba
   [junit4]   2> 1229400 T3730 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/control/data
   [junit4]   2> 1229401 T3730 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/control/data/index/
   [junit4]   2> 1229401 T3730 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1229401 T3730 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/control/data/index
   [junit4]   2> 1229402 T3730 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=39, maxMergeAtOnceExplicit=31, maxMergedSegmentMB=75.07421875, floorSegmentMB=1.3525390625, forceMergeDeletesPctAllowed=20.919694207225035, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1229403 T3730 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@21ab0515 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@47fa2a18)),segFN=segments_1,generation=1}
   [junit4]   2> 1229403 T3730 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1229403 T3730 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 1229405 T3730 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1229406 T3730 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1229406 T3730 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 1229406 T3730 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 1229407 T3730 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1229407 T3730 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1229407 T3730 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 1229408 T3730 oasup.UniqFieldsUpdateProcessorFactory.init WARN Use of the 'fields' init param in UniqFieldsUpdateProcessorFactory is deprecated, please use 'fieldName' (or another FieldMutatingUpdateProcessorFactory selector option) instead
   [junit4]   2> 1229408 T3730 oasup.UniqFieldsUpdateProcessorFactory.init Replacing 'fields' init param with (individual) 'fieldName' params
   [junit4]   2> 1229408 T3730 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 1229409 T3730 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1229409 T3730 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1229410 T3730 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1229410 T3730 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1229410 T3730 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1229410 T3730 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1229411 T3730 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1229411 T3730 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1229412 T3730 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 1229412 T3730 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 1229413 T3730 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1229413 T3730 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1229413 T3730 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 1229414 T3730 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 1229414 T3730 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 1229415 T3730 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1229415 T3730 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1229415 T3730 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1229416 T3730 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1229416 T3730 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1229417 T3730 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1229417 T3730 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1229417 T3730 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1229418 T3730 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1229418 T3730 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 1229419 T3730 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 1229420 T3730 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1229421 T3730 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 1229423 T3730 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1229424 T3730 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1229424 T3730 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=19, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6150124125500037]
   [junit4]   2> 1229425 T3730 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@21ab0515 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@47fa2a18)),segFN=segments_1,generation=1}
   [junit4]   2> 1229425 T3730 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1229426 T3730 oass.SolrIndexSearcher.<init> Opening Searcher@6ee1c17[collection1] main
   [junit4]   2> 1229426 T3730 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 1229479 T3730 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 1229496 T3731 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6ee1c17[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1229498 T3730 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1229499 T3720 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1229500 T3720 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1229501 T3720 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1229503 T3720 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40877
   [junit4]   2> 1229503 T3720 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1229504 T3720 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1229505 T3720 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr
   [junit4]   2> 1229505 T3720 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/'
   [junit4]   2> 1229513 T3720 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/solr.xml
   [junit4]   2> 1229547 T3720 oasc.CoreContainer.<init> New CoreContainer -796612920
   [junit4]   2> 1229547 T3720 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/]
   [junit4]   2> 1229549 T3720 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1229549 T3720 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1229550 T3720 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1229550 T3720 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1229551 T3720 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1229551 T3720 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1229552 T3720 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1229553 T3720 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1229553 T3720 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1229556 T3720 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1229557 T3720 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1229557 T3720 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1229561 T3742 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1
   [junit4]   2> 1229561 T3742 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/'
   [junit4]   2> 1229562 T3742 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 1229562 T3742 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 1229598 T3742 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 1229649 T3742 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1229650 T3742 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1229656 T3742 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1230002 T3742 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1230002 T3742 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1230003 T3742 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1230008 T3742 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1230010 T3742 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1230015 T3742 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1230019 T3742 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1230023 T3742 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1230023 T3742 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1230024 T3742 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1230024 T3742 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1230024 T3742 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1230025 T3742 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1230025 T3742 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1230025 T3742 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/shard0/data/
   [junit4]   2> 1230026 T3742 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@48da44ba
   [junit4]   2> 1230027 T3742 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/shard0/data
   [junit4]   2> 1230027 T3742 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/shard0/data/index/
   [junit4]   2> 1230027 T3742 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/shard0/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1230028 T3742 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/shard0/data/index
   [junit4]   2> 1230028 T3742 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=39, maxMergeAtOnceExplicit=31, maxMergedSegmentMB=75.07421875, floorSegmentMB=1.3525390625, forceMergeDeletesPctAllowed=20.919694207225035, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1230029 T3742 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1ac2fb29 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@15c91418)),segFN=segments_1,generation=1}
   [junit4]   2> 1230029 T3742 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1230030 T3742 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 1230032 T3742 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1230032 T3742 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1230033 T3742 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 1230033 T3742 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 1230033 T3742 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1230034 T3742 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1230034 T3742 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 1230034 T3742 oasup.UniqFieldsUpdateProcessorFactory.init WARN Use of the 'fields' init param in UniqFieldsUpdateProcessorFactory is deprecated, please use 'fieldName' (or another FieldMutatingUpdateProcessorFactory selector option) instead
   [junit4]   2> 1230035 T3742 oasup.UniqFieldsUpdateProcessorFactory.init Replacing 'fields' init param with (individual) 'fieldName' params
   [junit4]   2> 1230035 T3742 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 1230035 T3742 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1230036 T3742 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1230036 T3742 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1230036 T3742 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1230037 T3742 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1230037 T3742 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1230038 T3742 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1230038 T3742 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1230039 T3742 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 1230039 T3742 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 1230040 T3742 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1230040 T3742 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1230041 T3742 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 1230041 T3742 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 1230041 T3742 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 1230042 T3742 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1230042 T3742 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1230043 T3742 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1230043 T3742 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1230044 T3742 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1230044 T3742 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1230045 T3742 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1230045 T3742 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1230046 T3742 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1230046 T3742 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 1230047 T3742 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 1230048 T3742 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1230049 T3742 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 1230051 T3742 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1230052 T3742 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1230053 T3742 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=19, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6150124125500037]
   [junit4]   2> 1230053 T3742 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1ac2fb29 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@15c91418)),segFN=segments_1,generation=1}
   [junit4]   2> 1230053 T3742 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1230054 T3742 oass.SolrIndexSearcher.<init> Opening Searcher@91d000da[collection1] main
   [junit4]   2> 1230054 T3742 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 1230091 T3742 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 1230152 T3743 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@91d000da[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1230156 T3742 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1230157 T3720 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1230157 T3720 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> ASYNC  NEW_CORE C2383 name=collection1 org.apache.solr.core.SolrCore@42a2a7d
   [junit4]   2> 1230163 T3727 C2383 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[0 (1460388493974831104)]} 0 4
   [junit4]   2> ASYNC  NEW_CORE C2384 name=collection1 org.apache.solr.core.SolrCore@fa0180bd
   [junit4]   2> 1230167 T3740 C2384 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[0 (1460388493982171136)]} 0 1
   [junit4]   2> 1230171 T3728 C2383 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[1 (1460388493986365440)]} 0 1
   [junit4]   2> 1230175 T3741 C2384 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[1 (1460388493990559744)]} 0 1
   [junit4]   2> 1230177 T3729 C2383 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2 (1460388493993705472)]} 0 0
   [junit4]   2> 1230180 T3738 C2384 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[2 (1460388493996851200)]} 0 0
   [junit4]   2> 1230183 T3726 C2383 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[3 (1460388493998948352)]} 0 1
   [junit4]   2> 1230186 T3739 C2384 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[3 (1460388494002094080)]} 0 1
   [junit4]   2> 1230189 T3727 C2383 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[4 (1460388494005239808)]} 0 1
   [junit4]   2> 1230191 T3740 C2384 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[4 (1460388494008385536)]} 0 0
   [junit4]   2> 1230194 T3728 C2383 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[5 (1460388494011531264)]} 0 0
   [junit4]   2> 1230197 T3741 C2384 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[5 (1460388494014676992)]} 0 0
   [junit4]   2> 1230200 T3729 C2383 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[6 (1460388494017822720)]} 0 0
   [junit4]   2> 1230203 T3738 C2384 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[6 (1460388494019919872)]} 0 1
   [junit4]   2> 1230206 T3726 C2383 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[7 (1460388494023065600)]} 0 1
   [junit4]   2> 1230210 T3739 C2384 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[7 (1460388494028308480)]} 0 0
   [junit4]   2> 1230214 T3727 C2383 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[8 (1460388494032502784)]} 0 0
   [junit4]   2> 1230217 T3740 C2384 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[8 (1460388494034599936)]} 0 1
   [junit4]   2> 1230219 T3728 C2383 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[9 (1460388494037745664)]} 0 0
   [junit4]   2> 1230222 T3741 C2384 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[9 (1460388494039842816)]} 0 1
   [junit4]   2> 1230224 T3729 C2383 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1230228 T3729 C2383 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@21ab0515 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@47fa2a18)),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@21ab0515 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@47fa2a18)),segFN=segments_2,generation=2}
   [junit4]   2> 1230229 T3729 C2383 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 1230233 T3729 C2383 oass.SolrIndexSearcher.<init> Opening Searcher@b39fe3d5[collection1] main
   [junit4]   2> 1230234 T3729 C2383 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 1230236 T3731 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@b39fe3d5[collection1] main{StandardDirectoryReader(segments_2:3:nrt _0(4.8):c10)}
   [junit4]   2> 1230236 T3729 C2383 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={commit=true&softCommit=false&version=2&wt=javabin&waitSearcher=true} {commit=} 0 12
   [junit4]   2> 1230238 T3738 C2384 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1230242 T3738 C2384 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1ac2fb29 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@15c91418)),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1ac2fb29 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@15c91418)),segFN=segments_2,generation=2}
   [junit4]   2> 1230243 T3738 C2384 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 1230245 T3738 C2384 oass.SolrIndexSearcher.<init> Opening Searcher@24fe01a2[collection1] main
   [junit4]   2> 1230246 T3738 C2384 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 1230247 T3743 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@24fe01a2[collection1] main{StandardDirectoryReader(segments_2:3:nrt _0(4.8):c10)}
   [junit4]   2> 1230247 T3738 C2384 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin&waitSearcher=true&commit=true&softCommit=false} {commit=} 0 9
   [junit4]   2> 1230256 T3726 C2383 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=id:0&shards.qt=tvrh&tv.tf=true&tv=true&qt=tvrh&sort=id+desc&version=2&distrib=false&wt=javabin} hits=1 status=0 QTime=6 
   [junit4]   2> 1230260 T3740 C2384 oasc.SolrCore.execute [collection1] webapp= path=/select params={version=2&sort=id+desc&wt=javabin&rows=10&NOW=1392734998778&q=id:0&fl=id&tv=true&qt=tvrh&tv.tf=true&fsv=true&isShard=true&distrib=false&start=0&shard.url=http://127.0.0.1:40877/|[ff01::114]:33332/&shards.qt=tvrh} hits=1 status=0 QTime=0 
   [junit4]   2> 1230263 T3741 C2384 oasc.SolrCore.execute [collection1] webapp= path=/select params={version=2&isShard=true&qt=tvrh&tv=true&tv.tf=true&wt=javabin&distrib=false&NOW=1392734998778&ids=0&shards.qt=tvrh&q=id:0&shard.url=http://127.0.0.1:40877/|[ff01::114]:33332/} status=0 QTime=0 
   [junit4]   2> 1230264 T3739 C2384 oasc.SolrCore.execute [collection1] webapp= path=/select params={shards=http://127.0.0.1:40877/|[ff01::114]:33332/&q=id:0&shards.qt=tvrh&tv=true&qt=tvrh&tv.tf=true&version=2&sort=id+desc&wt=javabin} hits=1 status=0 QTime=6 
   [junit4]   2> 1230267 T3727 C2383 oasc.SolrCore.execute [collection1] webapp= path=/select params={distrib=false&tv.fl=test_basictv,test_offtv&wt=javabin&tv=true&qt=tvrh&tv.tf=true&version=2&sort=id+asc&q=id:0&shards.qt=tvrh&fl=*,score} hits=1 status=0 QTime=1 
   [junit4]   2> 1230271 T3740 C2384 oasc.SolrCore.execute [collection1] webapp= path=/select params={wt=javabin&version=2&sort=id+asc&NOW=1392734998789&q=id:0&fl=id,score&rows=10&distrib=false&start=0&tv.fl=test_basictv,test_offtv&tv=true&qt=tvrh&tv.tf=true&fsv=true&isShard=true&shard.url=http://127.0.0.1:40877/&shards.qt=tvrh} hits=1 status=0 QTime=0 
   [junit4]   2> 1230274 T3741 C2384 oasc.SolrCore.execute [collection1] webapp= path=/select params={version=2&wt=javabin&q=id:0&NOW=1392734998789&fl=*,score&isShard=true&tv=true&qt=tvrh&tv.tf=true&tv.fl=test_basictv,test_offtv&distrib=false&shards.qt=tvrh&ids=0&shard.url=http://127.0.0.1:40877/} status=0 QTime=0 
   [junit4]   2> 1230275 T3738 C2384 oasc.SolrCore.execute [collection1] webapp= path=/select params={shards=http://127.0.0.1:40877/&q=id:0&shards.qt=tvrh&fl=*,score&tv.fl=test_basictv,test_offtv&wt=javabin&version=2&sort=id+asc&tv=true&qt=tvrh&tv.tf=true} hits=1 status=0 QTime=6 
   [junit4]   2> 1230278 T3728 C2383 oasc.SolrCore.execute [collection1] webapp= path=/select params={wt=javabin&tv.fl=test_basictv&tv.fl=test_offtv&distrib=false&sort=id+asc&version=2&tv.tf=true&tv=true&qt=tvrh&fl=*,score&q=id:0&shards.qt=tvrh} hits=1 status=0 QTime=1 
   [junit4]   2> 1230284 T3740 C2384 oasc.SolrCore.execute [collection1] webapp= path=/select params={shard.url=http://127.0.0.1:40877/|[ff01::114]:33332/|[ff01::213]:33332/&shards.qt=tvrh&start=0&distrib=false&tv.fl=test_basictv&tv.fl=test_offtv&tv.tf=true&fsv=true&tv=true&qt=tvrh&isShard=true&fl=id,score&NOW=1392734998800&q=id:0&rows=10&wt=javabin&sort=id+asc&version=2} hits=1 status=0 QTime=0 
   [junit4]   2> 1230287 T3741 C2384 oasc.SolrCore.execute [collection1] webapp= path=/select params={shards.qt=tvrh&ids=0&shard.url=http://127.0.0.1:40877/|[ff01::114]:33332/|[ff01::213]:33332/&tv.fl=test_basictv&tv.fl=test_offtv&distrib=false&isShard=true&tv.tf=true&tv=true&qt=tvrh&fl=*,score&q=id:0&NOW=1392734998800&wt=javabin&version=2} status=0 QTime=1 
   [junit4]   2> 1230288 T3739 C2384 oasc.SolrCore.execute [collection1] webapp= path=/select params={shards=http://127.0.0.1:40877/|[ff01::114]:33332/|[ff01::213]:33332/&q=id:0&shards.qt=tvrh&fl=*,score&tv.fl=test_basictv&tv.fl=test_offtv&wt=javabin&version=2&sort=id+asc&tv=true&qt=tvrh&tv.tf=true} hits=1 status=0 QTime=8 
   [junit4]   2> 1230289 T3720 oas.BaseDistributedSearchTestCase.compareSolrResponses ERROR Mismatched responses:
   [junit4]   2> 	{responseHeader={status=0,QTime=8},response={numFound=1,start=0,maxScore=2.609438,docs=[SolrDocument{id=0, range_facet_si=0, range_facet_l=[0], range_facet_sl=[0], test_posofftv=[This is a title and another title], test_basictv=[This is a title and another title], test_notv=[This is a title and another title], test_postv=[This is a title and another title], test_offtv=[This is a title and another title], _version_=1460388493982171136, intDefault=42, timestamp=Tue Feb 18 05:49:58 AKST 2014, multiDefault=[muLti-Default], score=2.609438}]},termVectors={uniqueKeyFieldName=id,0={uniqueKey=0,test_basictv={anoth={tf=1},titl={tf=2}},test_offtv={anoth={tf=1},titl={tf=2}}}}}
   [junit4]   2> 	{responseHeader={status=0,QTime=1},response={numFound=1,start=0,maxScore=2.609438,docs=[SolrDocument{id=0, range_facet_si=0, range_facet_l=[0], range_facet_sl=[0], test_posofftv=[This is a title and another title], test_basictv=[This is a title and another title], test_notv=[This is a title and another title], test_postv=[This is a title and another title], test_offtv=[This is a title and another title], _version_=1460388493974831104, intDefault=42, timestamp=Tue Feb 18 05:49:58 AKST 2014, multiDefault=[muLti-Default], score=2.609438}]},termVectors={uniqueKeyFieldName=id,0={uniqueKey=0,test_offtv={anoth={tf=1},titl={tf=2}},test_basictv={anoth={tf=1},titl={tf=2}}}}}
   [junit4]   2> 1230310 T3720 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=-1567636093
   [junit4]   2> 1230311 T3720 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@42a2a7d
   [junit4]   2> 1230321 T3720 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=10,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=1285,transaction_logs_total_number=1}
   [junit4]   2> 1230322 T3720 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1230323 T3720 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1230324 T3720 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1230325 T3720 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1230326 T3720 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1230327 T3720 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/control/data/index [CachedDir<<refCount=0;path=./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/control/data/index;done=false>>]
   [junit4]   2> 1230328 T3720 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/control/data/index
   [junit4]   2> 1230328 T3720 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/control/data [CachedDir<<refCount=0;path=./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/control/data;done=false>>]
   [junit4]   2> 1230329 T3720 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/control/data
   [junit4]   2> 1230329 T3720 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 1230402 T3720 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=-796612920
   [junit4]   2> 1230403 T3720 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@fa0180bd
   [junit4]   2> 1230417 T3720 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=10,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=1285,transaction_logs_total_number=1}
   [junit4]   2> 1230418 T3720 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1230419 T3720 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1230424 T3720 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1230425 T3720 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1230426 T3720 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1230427 T3720 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/shard0/data [CachedDir<<refCount=0;path=./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/shard0/data;done=false>>]
   [junit4]   2> 1230428 T3720 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/shard0/data
   [junit4]   2> 1230429 T3720 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/shard0/data/index [CachedDir<<refCount=0;path=./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/shard0/data/index;done=false>>]
   [junit4]   2> 1230430 T3720 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.handler.component.TermVectorComponentDistributedTest-1392734997406/shard0/data/index
   [junit4]   2> 1230431 T3720 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 1230489 T3720 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TermVectorComponentDistributedTest -Dtests.method=testDistribSearch -Dtests.seed=C64D18B3B757CA71 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sr_RS_#Latn -Dtests.timezone=America/Yakutat -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 1.61s J1 | TermVectorComponentDistributedTest.testDistribSearch <<<
   [junit4]    > Throwable #1: junit.framework.AssertionFailedError: .termVectors.0.test_basictv!=test_offtv (unordered or missing)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([C64D18B3B757CA71:47AB96ABC008AA4D]:0)
   [junit4]    > 	at junit.framework.Assert.fail(Assert.java:50)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.compareSolrResponses(BaseDistributedSearchTestCase.java:843)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.compareResponses(BaseDistributedSearchTestCase.java:862)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:565)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:545)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:524)
   [junit4]    > 	at org.apache.solr.handler.component.TermVectorComponentDistributedTest.doTest(TermVectorComponentDistributedTest.java:137)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:876)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:780)
   [junit4]   2> 1230498 T3720 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Lucene46: {test_posofftv=PostingsFormat(name=FSTPulsing41), timestamp=PostingsFormat(name=MockSep), _version_=PostingsFormat(name=FSTPulsing41), range_facet_l=PostingsFormat(name=MockSep), intDefault=PostingsFormat(name=FSTPulsing41), id=PostingsFormat(name=SimpleText), multiDefault=PostingsFormat(name=MockSep), test_offtv=PostingsFormat(name=FSTPulsing41), range_facet_si=PostingsFormat(name=MockSep), test_postv=Lucene41(blocksize=128), range_facet_sl=PostingsFormat(name=SimpleText), test_basictv=PostingsFormat(name=SimpleText), test_notv=PostingsFormat(name=MockSep)}, docValues:{}, sim=DefaultSimilarity, locale=sr_RS_#Latn, timezone=America/Yakutat
   [junit4]   2> NOTE: Linux 3.8.0-35-generic x86/IBM Corporation 1.7.0 (32-bit)/cpus=8,threads=1,free=166791848,total=295305216
   [junit4]   2> NOTE: All tests run in this JVM: [RemoteQueryErrorTest, TestRTGBase, ParsingFieldUpdateProcessorsTest, IndexSchemaRuntimeFieldTest, TestNRTOpen, OverseerRolesTest, DocumentAnalysisRequestHandlerTest, TestDynamicFieldResource, TestFieldCollectionResource, TimeZoneUtilsTest, TestStressRecovery, TestCopyFieldCollectionResource, TestStressReorder, TestDistributedSearch, SpellingQueryConverterTest, TestAtomicUpdateErrorCases, HdfsCollectionsAPIDistributedZkTest, NumericFieldsTest, TestJmxIntegration, TestFieldTypeResource, DebugComponentTest, TestFiltering, TestSolrCoreProperties, AddBlockUpdateTest, TestSchemaVersionResource, TestQuerySenderNoQuery, TestCodecSupport, DefaultValueUpdateProcessorTest, CurrencyFieldXmlFileTest, TestReload, HighlighterTest, SortByFunctionTest, MigrateRouteKeyTest, TestFreeTextSuggestions, TestNumberUtils, BasicDistributedZk2Test, CurrencyFieldOpenExchangeTest, TestMultiCoreConfBootstrap, TestCoreDiscovery, TestCollationFieldDocValues, TestNonDefinedSimilarityFactory, TestSolrXmlPersistor, StatsComponentTest, SolrPluginUtilsTest, DistanceFunctionTest, FieldAnalysisRequestHandlerTest, SuggesterWFSTTest, BadIndexSchemaTest, CursorPagingTest, SpellCheckComponentTest, TestHashPartitioner, OpenExchangeRatesOrgProviderTest, TestArbitraryIndexDir, TestSchemaSimilarityResource, BJQParserTest, TestSimpleQParserPlugin, TestCollationField, ZkNodePropsTest, XsltUpdateRequestHandlerTest, RequiredFieldsTest, DocValuesMultiTest, TestSweetSpotSimilarityFactory, TestManagedSchemaFieldResource, PreAnalyzedFieldTest, HardAutoCommitTest, UniqFieldsUpdateProcessorFactoryTest, TestUpdate, TestDefaultSearchFieldResource, TestIndexSearcher, DirectUpdateHandlerTest, TestFuzzyAnalyzedSuggestions, SolrRequestParserTest, TestHighlightDedupGrouping, TestIndexingPerformance, SuggestComponentTest, TestStressUserVersions, FileUtilsTest, TestFoldingMultitermQuery, TestFileDictionaryLookup, ConvertedLegacyTest, TestHighFrequencyDictionaryFactory, TestOmitPositions, TestZkChroot, PathHierarchyTokenizerFactoryTest, CursorMarkTest, TestLazyCores, PingRequestHandlerTest, TestClassNameShortening, NotRequiredUniqueKeyTest, BasicDistributedZkTest, CollectionsAPIDistributedZkTest, OpenCloseCoreStressTest, OverseerTest, BasicZkTest, ClusterStateUpdateTest, TestRandomFaceting, TestFaceting, TestRecovery, TermVectorComponentDistributedTest]
   [junit4] Completed on J1 in 1.64s, 1 test, 1 failure <<< FAILURES!

[...truncated 43 lines...]
   [junit4] Suite: org.apache.solr.handler.component.DistributedQueryElevationComponentTest
   [junit4]   2> 1326592 T4701 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 1326592 T4701 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /nvic/s
   [junit4]   2> 1326596 T4701 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 1326597 T4701 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1326600 T4701 oejus.SslContextFactory.doStart Enabled Protocols [TLSv1] of [SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1326604 T4701 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:60904
   [junit4]   2> 1326606 T4701 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1326607 T4701 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1326607 T4701 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr
   [junit4]   2> 1326608 T4701 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/'
   [junit4]   2> 1326616 T4701 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/solr.xml
   [junit4]   2> 1326650 T4701 oasc.CoreContainer.<init> New CoreContainer 1078884461
   [junit4]   2> 1326651 T4701 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/]
   [junit4]   2> 1326652 T4701 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1326653 T4701 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: https
   [junit4]   2> 1326653 T4701 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1326654 T4701 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1326654 T4701 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1326655 T4701 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1326655 T4701 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1326656 T4701 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1326657 T4701 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1326661 T4701 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1326662 T4701 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1326662 T4701 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1326666 T4711 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1
   [junit4]   2> 1326666 T4711 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/'
   [junit4]   2> 1326667 T4711 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 1326667 T4711 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 1326698 T4711 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 1326751 T4711 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-elevate.xml
   [junit4]   2> 1326752 T4711 oass.IndexSchema.readSchema Reading Solr Schema from schema11.xml
   [junit4]   2> 1326756 T4711 oass.IndexSchema.readSchema [collection1] Schema name=example
   [junit4]   2> 1326891 T4711 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1326895 T4711 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1326899 T4711 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1326900 T4711 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1392735094914/control/data/
   [junit4]   2> 1326900 T4711 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2ad68826
   [junit4]   2> 1326901 T4711 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1392735094914/control/data
   [junit4]   2> 1326902 T4711 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1392735094914/control/data/index/
   [junit4]   2> 1326902 T4711 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1392735094914/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1326903 T4711 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1392735094914/control/data/index
   [junit4]   2> 1326903 T4711 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=13, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1326904 T4711 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@2fc35f47 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@dc98fc9e),segFN=segments_1,generation=1}
   [junit4]   2> 1326904 T4711 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1326906 T4711 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1326907 T4711 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1326907 T4711 oasc.RequestHandlers.initHandlersFromConfig created /elevate: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1326908 T4711 oasc.RequestHandlers.initHandlersFromConfig created /dataElevate: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 1326909 T4711 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1326910 T4711 oasc.RequestHandlers.initHandlersFromConfig WARN no default request handler is registered (either '/select' or 'standard')
   [junit4]   2> 1326911 T4711 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 1326912 T4711 oasc.SolrCore.initDeprecatedSupport WARN adding ShowFileRequestHandler with hidden files: [SOLRCONFIG-QUERYSENDER-NOQUERY.XML, SOLRCONFIG-TRANSFORMERS.XML, SCHEMA-REQUIRED-FIELDS.XML, DA_UTF8.XML, HUNSPELL-TEST.AFF, CURRENCY.XML, STOPWITHBOM.TXT, SCHEMA-DOCVALUES.XML, SCHEMA-LMDIRICHLET.XML, SOLRCONFIG-POSTINGSHIGHLIGHT.XML, BAD-SCHEMA-MISPLACED-ASTERISK-COPYFIELD-DEST-SHOULD-FAIL-TEST.XML, SCHEMA-NOT-REQUIRED-UNIQUE-KEY.XML, SCHEMA-BM25.XML, BAD-SCHEMA-CURRENCY-FT-BOGUS-CODE-IN-XML.XML, SCHEMA-LMJELINEKMERCER.XML, FREETEXTSUGGEST.TXT, SOLRCONFIG_CODEC.XML, SOLRCONFIG-MASTER1-KEEPONEBACKUP.XML, KEEP-1.TXT, MISSLEADING.EXTENSION.UPDATEPROCESSOR.JS.TXT, BAD-SCHEMA-CURRENCY-MULTIVALUED.XML, BAD-MP-SOLRCONFIG.XML, SOLRCONFIG-SLAVE1.XML, SOLRCONFIG-FUNCTIONQUERY.XML, BAD-SOLRCONFIG-MULTIPLE-CFS.XML, SOLRCONFIG-HIGHLIGHT.XML, BAD-SCHEMA-CODEC-GLOBAL-VS-FT-MISMATCH.XML, ENUMSCONFIG.XML, SOLRCONFIG-ELEVATE.XML, SCHEMA-CUSTOM-FIELD.XML, SCHEMA-MINIMAL.XML, SOLRCONFIG-MERGEPOLICY-DEFAULTS.XML, XSLT, SCHEMA-ADD-SCHEMA-FIELDS-UPDATE-PROCESSOR.XML, SYNONYMS.TXT, BAD-SCHEMA-NOT-INDEXED-BUT-TF.XML, SOLRCONFIG-MASTER2.XML, REGEX-BOOST-PROCESSOR-TEST.TXT, BAD-SCHEMA-SIM-GLOBAL-VS-FT-MISMATCH.XML, BAD-SCHEMA-NOT-INDEXED-BUT-POS.XML, BAD-SCHEMA-SWEETSPOT-PARTIAL-HYPERBOLIC.XML, STOP-2.TXT, SOLRCONFIG-SPELLCHECKCOMPONENT.XML, SOLRCONFIG-MERGEPOLICY-LEGACY.XML, BAD-SOLRCONFIG-MULTIPLE-INDEXCONFIGS.XML, MAPPING-ISOLATIN1ACCENT.TXT, SCHEMA-TFIDF.XML, STOPTYPES-1.TXT, SCHEMA-SIMPLEQPPLUGIN.XML, BAD-SCHEMA-DYNAMICFIELD-DEFAULT-VAL.XML, SOLRCONFIG-SCRIPT-UPDATEPROCESSOR.XML, SOLRCONFIG-MASTER.XML, SCHEMA-DISTRIBUTED-MISSING-SORT.XML, BAD-CURRENCY.XML, SOLRCONFIG-IMPLICITPROPERTIES.XML, SCHEMA11.XML, SOLRCONFIG-SNIPPET-PROCESSOR.XML, SCHEMA-REST.XML, SCHEMA-COLLATE.XML, SCHEMA-SIM.XML, BAD-SCHEMA-EXTERNAL-FILEFIELD.XML, SOLRCONFIG-REPEATER.XML, JASUGGEST.TXT, BAD-SCHEMA-NOT-INDEXED-BUT-NORMS.XML, SOLRCONFIG-TEST-MISC.XML, SCHEMA-SWEETSPOT.XML, SOLRCONFIG-MANAGED-SCHEMA.XML, SOLRCONFIG-DEEPPAGING.XML, SOLRCONFIG-REQHANDLER.INCL, BAD-SCHEMA-MULTIPLE-ASTERISK-COPYFIELD-DEST-SHOULD-FAIL-TEST.XML, STOPTYPES-2.TXT, SOLRCONFIG-SOLR-749.XML, BAD-SCHEMA-MISPLACED-ASTERISK-COPYFIELD-SOURCE-SHOULD-FAIL-TEST.XML, SOLRCONFIG-QUERY-PARSER-INIT.XML, SCHEMA-DOCVALUESMULTI.XML, SOLRCONFIG-CACHING.XML, BAD-SOLRCONFIG-MANAGED-SCHEMA-NAMED-SCHEMA.XML.XML, SOLRCONFIG-DEFAULTS.XML, BAD-SCHEMA-DYNAMICFIELD-REQUIRED.XML, HUNSPELL-TEST.DIC, SCHEMA-SORTS.XML, SOLRCONFIG.SNIPPET.RANDOMINDEXCONFIG.XML, BAD-SOLRCONFIG-SCHEMA-MUTABLE-BUT-NOT-MANAGED.XML, BAD-SOLRCONFIG-MISSING-SCRIPTFILE.XML, STOP-1.TXT, SCHEMA-DOCVALUESMISSING.XML, SOLRCONFIG-RESPONSE-LOG-COMPONENT.XML, SOLRCONFIG-WARMER.XML, ADDFIELDS.UPDATEPROCESSOR.JS, TRIVIAL.UPDATEPROCESSOR0.JS, SCHEMA-SPELLCHECKER.XML, BAD-SCHEMA-CURRENCY-FT-MULTIVALUED.XML, TRIVIAL.UPDATEPROCESSOR1.JS, SCHEMASURROUND.XML, SOLRCONFIG-MASTER1.XML, CONDITIONAL.UPDATEPROCESSOR.JS, SCHEMA-REPLICATION2.XML, BAD-SCHEMA-BOGUS-FIELD-PARAMETERS.XML, BAD-SCHEMA-SWEETSPOT-PARTIAL-BASELINE.XML, SOLRCONFIG-UPDATE-PROCESSOR-CHAINS.XML, BAD-SCHEMA-CURRENCY-FT-OER-NORATES.XML, SCHEMA-REPLICATION1.XML, SOLRCONFIG-MASTER3.XML, SOLRCONFIG-NOOPREGEN.XML, WDFTYPES.TXT, STOPWORDS.TXT, SOLRCONFIG-DELPOLICY1.XML, SCHEMA-DFR.XML, OPEN-EXCHANGE-RATES.JSON, SOLRCONFIG-INDEXCONFIG.XML, BAD-SCHEMA-DUP-DYNAMICFIELD.XML, SOLRCONFIG-TIEREDMERGEPOLICY.XML, SCHEMA-CUSTOMFIELD.XML, SCHEMA_CODEC.XML, SCHEMA-XINCLUDE.XML, SCHEMA-DOCVALUESFACETING.XML, SCHEMA-SYNONYM-TOKENIZER.XML, SCHEMA-POSTINGSHIGHLIGHT.XML, SCHEMA-SNIPPET-TYPE.XML, SCHEMA-SPATIAL.XML, ANALYZINGINFIXSUGGEST.TXT, SCHEMA-ID-AND-VERSION-FIELDS-ONLY.XML, OLD_SYNONYMS.TXT, SCHEMA-COLLATE-DV.XML, SCHEMA-NUMERIC.XML, SOLRCONFIG-EXTERNALVERSIONCONSTRAINT.XML, SOLRCONFIG-PARSING-UPDATE-PROCESSOR-CHAINS.XML, SOLRCONFIG-COMPONENTS-NAME.XML, BAD-SOLRCONFIG-INVALID-SCRIPTFILE.XML, SOLRCONFIG-NOCACHE.XML, SCHEMA-COPYFIELD-TEST.XML, SOLRCONFIG-LOGMERGEPOLICY.XML, BAD-SCHEMA-UNIQUEKEY-USES-DEFAULT.XML, SCHEMA-CHARFILTERS.XML, SCHEMA-TINY.XML, PROTWORDS.TXT, SOLRCONFIG-TLOG.XML, SCHEMA-BEHAVIOR.XML, SCHEMA-FOLDING.XML, SOLRCONFIG-SPELLCHECKER.XML, COMPOUNDDICTIONARY.TXT, BAD-SCHEMA-DUP-FIELD.XML, SOLRCONFIG-TERMINDEX.XML, SOLRCONFIG-BASIC.XML, BAD-SCHEMA-UNIQUEKEY-IS-COPYFIELD-DEST.XML, SOLRCONFIG_PERF.XML, STOP-SNOWBALL.TXT, SOLRCONFIG-SUGGESTERCOMPONENT.XML, BAD-SCHEMA-BOGUS-ANALYSIS-PARAMETERS.XML, BAD-SCHEMA-CURRENCY-DYNAMIC-MULTIVALUED.XML, BAD-SOLRCONFIG-MULTIPLE-DIRFACTORY.XML, BAD-SCHEMA-DUP-FIELDTYPE.XML, BAD-SCHEMA-ANALYZER-CLASS-AND-NESTED.XML, BAD-SCHEMA-MULTIPLE-ASTERISK-COPYFIELD-SOURCE-SHOULD-FAIL-TEST.XML, SOLRCONFIG-BASIC-LUCENEVERSION31.XML, THROW.ERROR.ON.ADD.UPDATEPROCESSOR.JS, BAD-SOLRCONFIG-BOGUS-SCRIPTENGINE-NAME.XML, BAD-SOLRCONFIG-WARMER-NO-REOPEN.XML, SCHEMA-CLASS-NAME-SHORTENING-ON-SERIALIZATION.XML, SOLRCONFIG-DELPOLICY2.XML, DA_COMPOUNDDICTIONARY.TXT, BAD-SCHEMA-SWEETSPOT-PARTIAL-NORMS.XML, SOLRCONFIG-MINIMAL.XML, BAD_SOLRCONFIG.XML, FUZZYSUGGEST.TXT, SCHEMA-LUCENEMATCHVERSION.XML, SOLRCONFIG-SLAVE.XML, SCHEMA-IB.XML, SOLRCONFIG-LAZYWRITER.XML, SCHEMA-COLLATEFILTER.XML, SCHEMA15.XML, SCHEMA-ONE-FIELD-NO-DYNAMIC-FIELD-UNIQUE-KEY.XML, SOLRCONFIG-INFOSTREAM-LOGGING.XML, SOLRCONFIG-COLLAPSEQPARSER.XML, SCHEMA-SNIPPET-TYPES.INCL, STOPWORDSWRONGENCODING.TXT, BAD-SCHEMA-NON-GLOB-COPYFIELD-SOURCE-MATCHING-NOTHING-SHOULD-FAIL-TEST.XML, BAD-SCHEMA-CURRENCY-FT-BOGUS-DEFAULT-CODE.XML, SCHEMA-PHRASESUGGEST.XML, SOLRCONFIG-PHRASESUGGEST.XML, SCHEMA-REVERSED.XML, PHRASESUGGEST.TXT, SCHEMA-EFF.XML, MISSING.FUNCTIONS.UPDATEPROCESSOR.JS, SCHEMA-SNIPPET-FIELD.XML, BAD-SCHEMA-NONTEXT-ANALYZER.XML, SOLRCONFIG-XINCLUDE.XML, BAD-SCHEMA-SWEETSPOT-BOTH-TF.XML, FRENCHARTICLES.TXT, BLENDEDINFIXSUGGEST.TXT, KEEP-2.TXT, SCHEMA-ONE-FIELD-NO-DYNAMIC-FIELD.XML, SOLRCONFIG-QUERYSENDER.XML, SCHEMA-REST-LUCENE-MATCH-VERSION.XML, STEMDICT.TXT, SCHEMA-STOP-KEEP.XML, SOLRCONFIG-ADD-SCHEMA-FIELDS-UPDATE-PROCESSOR-CHAINS.XML, BAD-SCHEMA-OMIT-TF-BUT-NOT-POS.XML, SOLRCONFIG-SOLCOREPROPERTIES.XML, BAD-SCHEMA-UNIQUEKEY-MULTIVALUED.XML, SCHEMA-ENUMS.XML, BAD-SOLRCONFIG-UNEXPECTED-SCHEMA-ATTRIBUTE.XML, ELEVATE.XML, BAD-SCHEMA-UNSUPPORTED-DOCVALUES.XML, SCHEMA-TRIE.XML, SCHEMA-BINARYFIELD.XML, SOLRCONFIG-ALTDIRECTORY.XML, SCHEMA12.XML, HYPHENATION.DTD]
   [junit4]   2> 1326914 T4711 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1326914 T4711 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1326915 T4711 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=97.2216796875, floorSegmentMB=1.3271484375, forceMergeDeletesPctAllowed=7.240384447564098, segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1326916 T4711 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@2fc35f47 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@dc98fc9e),segFN=segments_1,generation=1}
   [junit4]   2> 1326916 T4711 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1326917 T4711 oass.SolrIndexSearcher.<init> Opening Searcher@19aee66a[collection1] main
   [junit4]   2> 1326917 T4711 oashc.QueryElevationComponent.inform Loading QueryElevation from: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/elevate.xml
   [junit4]   2> 1326927 T4711 oashc.QueryElevationComponent.inform Loading QueryElevation from: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/elevate.xml
   [junit4]   2> 1326937 T4712 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@19aee66a[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1326938 T4711 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1326939 T4701 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 1326939 T4701 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1326941 T4701 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1326943 T4701 oejus.SslContextFactory.doStart Enabled Protocols [TLSv1] of [SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1326944 T4701 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:43841
   [junit4]   2> 1326946 T4701 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1326946 T4701 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1326947 T4701 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr
   [junit4]   2> 1326948 T4701 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/'
   [junit4]   2> 1326955 T4701 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/solr.xml
   [junit4]   2> 1326990 T4701 oasc.CoreContainer.<init> New CoreContainer 1655269720
   [junit4]   2> 1326990 T4701 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/]
   [junit4]   2> 1326992 T4701 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1326992 T4701 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: https
   [junit4]   2> 1326993 T4701 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1326994 T4701 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1326994 T4701 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1326995 T4701 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1326995 T4701 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1326996 T4701 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1326996 T4701 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1327001 T4701 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1327002 T4701 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1327003 T4701 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1327006 T4723 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1
   [junit4]   2> 1327006 T4723 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/'
   [junit4]   2> 1327007 T4723 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.

[...truncated too long message...]

ache.solr.update.processor.SignatureUpdateProcessorFactoryTest.testDupeDetection(SignatureUpdateProcessorFactoryTest.java:119)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:780)
   [junit4]   2> 1331100 T4757 oas.SolrTestCaseJ4.setUp ###Starting testNonStringFieldsValues
   [junit4]   2> 1331100 T4757 C2079 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 1331101 T4757 C2079 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {deleteByQuery=*:*} 0 1
   [junit4]   2> 1331103 T4757 C2079 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1331104 T4757 C2079 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@2fa85e79 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@963e4408)),segFN=segments_12,generation=38}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@2fa85e79 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@963e4408)),segFN=segments_13,generation=39}
   [junit4]   2> 1331105 T4757 C2079 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 39
   [junit4]   2> 1331106 T4757 C2079 oass.SolrIndexSearcher.<init> Opening Searcher@d08754da[collection1] main
   [junit4]   2> 1331107 T4757 C2079 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 1331108 T4759 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@d08754da[collection1] main{StandardDirectoryReader(segments_13:163:nrt)}
   [junit4]   2> 1331109 T4757 C2079 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={} {commit=} 0 6
   [junit4]   2> 1331112 T4757 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1331114 T4757 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1331116 T4757 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@2fa85e79 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@963e4408)),segFN=segments_13,generation=39}
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@2fa85e79 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@963e4408)),segFN=segments_14,generation=40}
   [junit4]   2> 1331116 T4757 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 40
   [junit4]   2> 1331118 T4757 oass.SolrIndexSearcher.<init> Opening Searcher@cbb6d722[collection1] main
   [junit4]   2> 1331118 T4757 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 1331121 T4759 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@cbb6d722[collection1] main{StandardDirectoryReader(segments_14:165:nrt _1f(4.8):C5/1:delGen=1)}
   [junit4]   2> 1331121 T4757 oas.SolrTestCaseJ4.tearDown ###Ending testNonStringFieldsValues
   [junit4]   2> 1331123 T4757 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 1331124 T4757 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=928948566
   [junit4]   2> 1331124 T4757 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@ab3e2629
   [junit4]   2> 1331137 T4757 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=42,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=253,cumulative_deletesById=0,cumulative_deletesByQuery=6,cumulative_errors=0}
   [junit4]   2> 1331138 T4757 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1331138 T4757 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1331139 T4757 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1331139 T4757 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1331140 T4757 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1331141 T4757 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-SignatureUpdateProcessorFactoryTest-1392735097833/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-SignatureUpdateProcessorFactoryTest-1392735097833/index;done=false>>]
   [junit4]   2> 1331141 T4757 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-SignatureUpdateProcessorFactoryTest-1392735097833/index
   [junit4]   2> 1331142 T4757 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-SignatureUpdateProcessorFactoryTest-1392735097833 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-SignatureUpdateProcessorFactoryTest-1392735097833;done=false>>]
   [junit4]   2> 1331142 T4757 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-SignatureUpdateProcessorFactoryTest-1392735097833
   [junit4]   2> NOTE: test params are: codec=Lucene45, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=sr_ME, timezone=America/Goose_Bay
   [junit4]   2> NOTE: Linux 3.8.0-35-generic x86/IBM Corporation 1.7.0 (32-bit)/cpus=8,threads=1,free=49574792,total=339607552
   [junit4]   2> NOTE: All tests run in this JVM: [OverseerCollectionProcessorTest, TestCloudManagedSchemaAddField, SoftAutoCommitTest, TestWriterPerf, SOLR749Test, LeaderElectionTest, TestMergePolicyConfig, TermVectorComponentTest, TermsComponentTest, TestSystemIdResolver, TestElisionMultitermQuery, ResponseLogComponentTest, TestExtendedDismaxParser, TestDocumentBuilder, ShowFileRequestHandlerTest, TestBM25SimilarityFactory, HdfsLockFactoryTest, TestDistribDocBasedVersion, TestFieldResource, QueryResultKeyTest, FastVectorHighlighterTest, JsonLoaderTest, ExternalFileFieldSortTest, TestRandomDVFaceting, SuggesterTest, TestBadConfig, HdfsUnloadDistributedZkTest, PeerSyncTest, SimpleFacetsTest, QueryParsingTest, MBeansHandlerTest, TestLFUCache, DeleteShardTest, DistributedQueryComponentCustomSortTest, SpellPossibilityIteratorTest, DateMathParserTest, TestCollationKeyRangeQueries, ShardSplitTest, CachingDirectoryFactoryTest, TestRecoveryHdfs, TestInfoStreamLogging, CoreAdminHandlerTest, UpdateRequestProcessorFactoryTest, TestLRUCache, TestNonNRTOpen, TestLMJelinekMercerSimilarityFactory, BasicFunctionalityTest, TestReplicationHandler, ShardRoutingTest, TestFieldTypeCollectionResource, HdfsSyncSliceTest, TestOverriddenPrefixQueryForCustomFieldType, ReturnFieldsTest, TestSolrQueryParserResource, SliceStateUpdateTest, TestSolrXml, TestSchemaResource, TestRemoteStreaming, TestSearchPerf, TestPostingsSolrHighlighter, TestSchemaNameResource, BlockCacheTest, RegexBoostProcessorTest, DirectUpdateHandlerOptimizeTest, SearchHandlerTest, PolyFieldTest, TestAnalyzeInfixSuggestions, TestSuggestSpellingConverter, TestStressVersions, TestGroupingSearch, AutoCommitTest, CoreMergeIndexesAdminHandlerTest, TestInitQParser, ScriptEngineTest, IndexSchemaTest, FileBasedSpellCheckerTest, AnalysisAfterCoreReloadTest, WordBreakSolrSpellCheckerTest, ChangedSchemaMergeTest, SliceStateTest, TestTrie, CopyFieldTest, ResourceLoaderTest, TestShardHandlerFactory, QueryElevationComponentTest, TestDynamicFieldCollectionResource, ChaosMonkeySafeLeaderTest, UnloadDistributedZkTest, SyncSliceTest, LeaderElectionIntegrationTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest, ZkSolrClientTest, ZkCLITest, ShardRoutingCustomTest, TestDistributedGrouping, DistributedSpellCheckComponentTest, TestRealTimeGet, TestJoin, SolrCoreTest, SolrCmdDistributorTest, TestFunctionQuery, DistributedQueryElevationComponentTest, SimplePostToolTest, SignatureUpdateProcessorFactoryTest]
   [junit4] Completed on J0 in 1.66s, 6 tests, 2 failures <<< FAILURES!

[...truncated 451 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:459: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:452: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1308: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:941: There were test failures: 362 suites, 1563 tests, 4 failures, 36 ignored (8 assumptions)

Total time: 55 minutes 54 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_51) - Build # 9408 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9408/
Java: 32bit/jdk1.7.0_51 -server -XX:+UseParallelGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([B275C9688947A547]:0)


FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([B275C9688947A547]:0)




Build Log:
[...truncated 11210 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> 622015 T1619 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 622015 T1619 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 622019 T1619 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeySafeLeaderTest-1392721576617
   [junit4]   2> 622019 T1619 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 622020 T1620 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 622120 T1619 oasc.ZkTestServer.run start zk server on port:52473
   [junit4]   2> 622121 T1619 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 622220 T1626 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1076396 name:ZooKeeperConnection Watcher:127.0.0.1:52473 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 622220 T1619 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 622221 T1619 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 622228 T1619 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 622230 T1628 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ce6123 name:ZooKeeperConnection Watcher:127.0.0.1:52473/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 622230 T1619 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 622230 T1619 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 622236 T1619 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 622241 T1619 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 622244 T1619 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 622249 T1619 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 622250 T1619 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 622258 T1619 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 622258 T1619 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 622265 T1619 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 622266 T1619 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 622271 T1619 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 622271 T1619 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 622276 T1619 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 622277 T1619 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 622282 T1619 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 622283 T1619 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 622287 T1619 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 622288 T1619 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 622293 T1619 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 622294 T1619 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 622298 T1619 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 622299 T1619 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 622304 T1619 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 622305 T1619 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 622310 T1619 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 622310 T1619 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 622317 T1619 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 622319 T1630 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ff2649 name:ZooKeeperConnection Watcher:127.0.0.1:52473/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 622319 T1619 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 622407 T1619 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 622410 T1619 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 622411 T1619 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:60884
   [junit4]   2> 622413 T1619 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 622413 T1619 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 622413 T1619 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392721576921
   [junit4]   2> 622413 T1619 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392721576921/'
   [junit4]   2> 622423 T1619 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392721576921/solr.xml
   [junit4]   2> 622454 T1619 oasc.CoreContainer.<init> New CoreContainer 17460510
   [junit4]   2> 622454 T1619 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392721576921/]
   [junit4]   2> 622455 T1619 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 622456 T1619 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 622456 T1619 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 622456 T1619 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 622456 T1619 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 622457 T1619 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 622457 T1619 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 622457 T1619 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 622457 T1619 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 622461 T1619 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 622461 T1619 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 622461 T1619 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 622461 T1619 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52473/solr
   [junit4]   2> 622462 T1619 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 622463 T1619 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 622464 T1641 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b0a870 name:ZooKeeperConnection Watcher:127.0.0.1:52473 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 622464 T1619 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 622466 T1619 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 622467 T1643 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@534cae name:ZooKeeperConnection Watcher:127.0.0.1:52473/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 622467 T1619 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 622468 T1619 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 622471 T1619 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 622475 T1619 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 622476 T1619 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60884_
   [junit4]   2> 622477 T1619 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60884_
   [junit4]   2> 622479 T1619 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 622481 T1619 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 622483 T1619 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:60884_
   [junit4]   2> 622483 T1619 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 622485 T1619 oasc.Overseer.start Overseer (id=91273401245302788-127.0.0.1:60884_-n_0000000000) starting
   [junit4]   2> 622488 T1619 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 622492 T1645 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 622492 T1619 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 622493 T1645 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 622495 T1619 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 622497 T1619 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 622500 T1644 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 622502 T1646 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 622503 T1646 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 622504 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 622504 T1646 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 622505 T1644 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 622505 T1644 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:60884",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:60884_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 622506 T1644 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 622506 T1644 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 622507 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 622508 T1643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 623504 T1646 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 623504 T1646 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392721576921/collection1
   [junit4]   2> 623505 T1646 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 623505 T1646 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 623506 T1646 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 623507 T1646 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 623507 T1646 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392721576921/collection1/'
   [junit4]   2> 623507 T1646 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392721576921/collection1/lib/classes/' to classloader
   [junit4]   2> 623507 T1646 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392721576921/collection1/lib/README' to classloader
   [junit4]   2> 623535 T1646 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 623565 T1646 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 623566 T1646 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 623572 T1646 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 623835 T1646 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 623837 T1646 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 623838 T1646 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 623842 T1646 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 623860 T1646 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 623866 T1646 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392721576921/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/control/data/
   [junit4]   2> 623866 T1646 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@121387e
   [junit4]   2> 623867 T1646 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/control/data
   [junit4]   2> 623867 T1646 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/control/data/index/
   [junit4]   2> 623867 T1646 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 623868 T1646 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/control/data/index
   [junit4]   2> 623868 T1646 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=18.9658203125, floorSegmentMB=2.013671875, forceMergeDeletesPctAllowed=28.875788144137676, segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3897341227826966
   [junit4]   2> 623872 T1646 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 623872 T1646 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 623874 T1646 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 623874 T1646 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 623874 T1646 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 623875 T1646 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 623875 T1646 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 623875 T1646 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 623875 T1646 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 623876 T1646 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 623876 T1646 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 623876 T1646 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 623876 T1646 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 623877 T1646 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 623877 T1646 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 623877 T1646 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 623877 T1646 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 623878 T1646 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 623880 T1646 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 623882 T1646 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 623882 T1646 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 623883 T1646 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1000120485, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 623883 T1646 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 623883 T1646 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 623884 T1646 oass.SolrIndexSearcher.<init> Opening Searcher@a9fe08[collection1] main
   [junit4]   2> 623885 T1647 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@a9fe08[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 623886 T1646 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 623887 T1619 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 623892 T1619 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 623891 T1650 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:60884 collection:control_collection shard:shard1
   [junit4]   2> 623893 T1650 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 623894 T1619 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 623895 T1652 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69083c name:ZooKeeperConnection Watcher:127.0.0.1:52473/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 623896 T1619 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 623897 T1619 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 623899 T1650 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 623899 T1619 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 623900 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 623901 T1650 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 623901 T1650 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C592 name=collection1 org.apache.solr.core.SolrCore@1a84cb9 url=https://127.0.0.1:60884/collection1 node=127.0.0.1:60884_ C592_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:60884, core=collection1, node_name=127.0.0.1:60884_}
   [junit4]   2> 623901 T1650 C592 P60884 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:60884/collection1/
   [junit4]   2> 623902 T1644 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 623902 T1650 C592 P60884 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 623902 T1650 C592 P60884 oasc.SyncStrategy.syncToMe https://127.0.0.1:60884/collection1/ has no replicas
   [junit4]   2> 623902 T1650 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:60884/collection1/ shard1
   [junit4]   2> 623903 T1650 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 623904 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 623907 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 623910 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 623971 T1619 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 623971 T1619 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 623974 T1619 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 623975 T1619 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:55496
   [junit4]   2> 623976 T1619 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 623977 T1619 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 623977 T1619 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392721578497
   [junit4]   2> 623977 T1619 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392721578497/'
   [junit4]   2> 623987 T1619 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392721578497/solr.xml
   [junit4]   2> 624013 T1643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 624014 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 624022 T1619 oasc.CoreContainer.<init> New CoreContainer 30476929
   [junit4]   2> 624022 T1619 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392721578497/]
   [junit4]   2> 624023 T1619 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 624023 T1619 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 624024 T1619 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 624024 T1619 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 624024 T1619 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 624024 T1619 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 624025 T1619 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 624025 T1619 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 624025 T1619 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 624029 T1619 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 624029 T1619 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 624029 T1619 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 624030 T1619 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52473/solr
   [junit4]   2> 624030 T1619 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 624031 T1619 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 624032 T1663 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13023e3 name:ZooKeeperConnection Watcher:127.0.0.1:52473 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 624032 T1619 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 624034 T1619 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 624035 T1665 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9c00ea name:ZooKeeperConnection Watcher:127.0.0.1:52473/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 624036 T1619 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 624039 T1619 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 624058 T1650 oasc.ZkController.register We are https://127.0.0.1:60884/collection1/ and leader is https://127.0.0.1:60884/collection1/
   [junit4]   2> 624058 T1650 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:60884
   [junit4]   2> 624059 T1650 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 624059 T1650 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 624059 T1650 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 624060 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 624060 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 624060 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 624060 T1650 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 624062 T1644 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 624062 T1644 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:60884",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:60884_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 624064 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 624166 T1643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 624166 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 624166 T1665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 625042 T1619 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55496_
   [junit4]   2> 625044 T1619 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55496_
   [junit4]   2> 625048 T1665 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 625048 T1643 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 625048 T1652 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 625056 T1666 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 625056 T1666 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 625057 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 625057 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 625057 T1666 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 625058 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 625059 T1644 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 625059 T1644 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:55496",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:55496_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 625060 T1644 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 625060 T1644 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 625063 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 625165 T1665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 625166 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 625166 T1643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 626058 T1666 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 626058 T1666 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392721578497/collection1
   [junit4]   2> 626058 T1666 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 626059 T1666 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 626059 T1666 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 626060 T1666 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 626060 T1666 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392721578497/collection1/'
   [junit4]   2> 626061 T1666 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392721578497/collection1/lib/classes/' to classloader
   [junit4]   2> 626061 T1666 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392721578497/collection1/lib/README' to classloader
   [junit4]   2> 626088 T1666 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 626119 T1666 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 626121 T1666 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 626126 T1666 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 626391 T1666 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 626393 T1666 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 626394 T1666 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 626397 T1666 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 626416 T1666 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 626416 T1666 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392721578497/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty1/
   [junit4]   2> 626416 T1666 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@121387e
   [junit4]   2> 626417 T1666 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty1
   [junit4]   2> 626417 T1666 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty1/index/
   [junit4]   2> 626417 T1666 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 626417 T1666 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty1/index
   [junit4]   2> 626418 T1666 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=18.9658203125, floorSegmentMB=2.013671875, forceMergeDeletesPctAllowed=28.875788144137676, segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3897341227826966
   [junit4]   2> 626420 T1666 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 626420 T1666 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 626422 T1666 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 626422 T1666 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 626423 T1666 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 626423 T1666 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 626423 T1666 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 626423 T1666 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 626424 T1666 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 626424 T1666 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 626424 T1666 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 626424 T1666 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 626425 T1666 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 626425 T1666 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 626425 T1666 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 626426 T1666 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 626426 T1666 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 626426 T1666 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 626428 T1666 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 626430 T1666 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 626431 T1666 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 626431 T1666 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1000120485, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 626432 T1666 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 626432 T1666 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 626433 T1666 oass.SolrIndexSearcher.<init> Opening Searcher@ece25a[collection1] main
   [junit4]   2> 626435 T1667 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@ece25a[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 626437 T1666 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 626437 T1619 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 626438 T1619 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 626437 T1670 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:55496 collection:collection1 shard:shard3
   [junit4]   2> 626439 T1670 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 626444 T1670 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 626446 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 626446 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 626446 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 626447 T1670 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 626447 T1670 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C593 name=collection1 org.apache.solr.core.SolrCore@c2a8ca url=https://127.0.0.1:55496/collection1 node=127.0.0.1:55496_ C593_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:55496, core=collection1, node_name=127.0.0.1:55496_}
   [junit4]   2> 626447 T1670 C593 P55496 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:55496/collection1/
   [junit4]   2> 626447 T1670 C593 P55496 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 626447 T1670 C593 P55496 oasc.SyncStrategy.syncToMe https://127.0.0.1:55496/collection1/ has no replicas
   [junit4]   2> 626447 T1644 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 626447 T1670 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:55496/collection1/ shard3
   [junit4]   2> 626448 T1670 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 626452 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 626454 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 626457 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 626550 T1619 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 626551 T1619 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 626554 T1619 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 626555 T1619 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:36580
   [junit4]   2> 626557 T1619 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 626557 T1619 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 626557 T1619 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392721581038
   [junit4]   2> 626557 T1619 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392721581038/'
   [junit4]   2> 626559 T1643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 626559 T1665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 626559 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 626569 T1619 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392721581038/solr.xml
   [junit4]   2> 626601 T1619 oasc.CoreContainer.<init> New CoreContainer 6419595
   [junit4]   2> 626601 T1619 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392721581038/]
   [junit4]   2> 626602 T1619 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 626602 T1619 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 626603 T1619 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 626603 T1619 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 626603 T1619 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 626603 T1619 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 626604 T1619 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 626604 T1619 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 626604 T1619 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 626605 T1670 oasc.ZkController.register We are https://127.0.0.1:55496/collection1/ and leader is https://127.0.0.1:55496/collection1/
   [junit4]   2> 626605 T1670 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:55496
   [junit4]   2> 626605 T1670 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 626606 T1670 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 626606 T1670 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 626607 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 626607 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 626607 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 626608 T1670 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 626608 T1619 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 626609 T1619 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 626609 T1644 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 626609 T1619 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 626610 T1619 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52473/solr
   [junit4]   2> 626610 T1644 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:55496",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:55496_",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 626610 T1619 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 626611 T1619 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 626612 T1681 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@906624 name:ZooKeeperConnection Watcher:127.0.0.1:52473 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 626612 T1619 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 626613 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 626614 T1619 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 626615 T1683 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2d631a name:ZooKeeperConnection Watcher:127.0.0.1:52473/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 626615 T1619 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 626619 T1619 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 626716 T1683 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 626716 T1643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 626716 T1665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 626716 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 627622 T1619 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36580_
   [junit4]   2> 627623 T1619 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36580_
   [junit4]   2> 627625 T1652 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 627625 T1643 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 627625 T1665 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 627625 T1683 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 627632 T1684 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 627633 T1684 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 627634 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 627634 T1684 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 627634 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 627634 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 627636 T1644 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 627636 T1644 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:36580",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:36580_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 627637 T1644 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 627637 T1644 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 627639 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 627741 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 627741 T1643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 627741 T1665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 627741 T1683 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 628634 T1684 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 628635 T1684 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392721581038/collection1
   [junit4]   2> 628635 T1684 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 628635 T1684 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 628636 T1684 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 628637 T1684 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 628637 T1684 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392721581038/collection1/'
   [junit4]   2> 628637 T1684 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392721581038/collection1/lib/classes/' to classloader
   [junit4]   2> 628638 T1684 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392721581038/collection1/lib/README' to classloader
   [junit4]   2> 628666 T1684 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 628704 T1684 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 628705 T1684 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 628711 T1684 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 628977 T1684 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 628979 T1684 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 628981 T1684 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 628984 T1684 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 629004 T1684 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 629004 T1684 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392721581038/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty2/
   [junit4]   2> 629005 T1684 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@121387e
   [junit4]   2> 629005 T1684 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty2
   [junit4]   2> 629005 T1684 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty2/index/
   [junit4]   2> 629006 T1684 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 629006 T1684 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty2/index
   [junit4]   2> 629006 T1684 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=18.9658203125, floorSegmentMB=2.013671875, forceMergeDeletesPctAllowed=28.875788144137676, segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3897341227826966
   [junit4]   2> 629009 T1684 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 629009 T1684 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 629011 T1684 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 629012 T1684 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 629012 T1684 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 629012 T1684 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 629012 T1684 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 629012 T1684 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 629013 T1684 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 629013 T1684 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 629013 T1684 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 629013 T1684 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 629014 T1684 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 629014 T1684 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 629014 T1684 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 629015 T1684 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 629015 T1684 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 629015 T1684 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 629017 T1684 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 629019 T1684 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 629020 T1684 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 629020 T1684 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1000120485, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 629021 T1684 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 629021 T1684 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 629021 T1684 oass.SolrIndexSearcher.<init> Opening Searcher@3900b4[collection1] main
   [junit4]   2> 629024 T1685 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3900b4[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 629025 T1684 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 629029 T1619 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 629030 T1619 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 629029 T1688 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:36580 collection:collection1 shard:shard1
   [junit4]   2> 629033 T1688 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 629038 T1688 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 629039 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 629039 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 629040 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 629040 T1688 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 629040 T1688 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C594 name=collection1 org.apache.solr.core.SolrCore@180ad17 url=https://127.0.0.1:36580/collection1 node=127.0.0.1:36580_ C594_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:36580, core=collection1, node_name=127.0.0.1:36580_}
   [junit4]   2> 629040 T1688 C594 P36580 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:36580/collection1/
   [junit4]   2> 629040 T1688 C594 P36580 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 629041 T1688 C594 P36580 oasc.SyncStrategy.syncToMe https://127.0.0.1:36580/collection1/ has no replicas
   [junit4]   2> 629041 T1644 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 629041 T1688 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:36580/collection1/ shard1
   [junit4]   2> 629041 T1688 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 629043 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 629049 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 629053 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 629104 T1619 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 629105 T1619 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 629107 T1619 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 629108 T1619 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:45441
   [junit4]   2> 629110 T1619 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 629110 T1619 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 629110 T1619 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392721583629
   [junit4]   2> 629111 T1619 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392721583629/'
   [junit4]   2> 629121 T1619 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392721583629/solr.xml
   [junit4]   2> 629153 T1619 oasc.CoreContainer.<init> New CoreContainer 7145772
   [junit4]   2> 629153 T1619 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392721583629/]
   [junit4]   2> 629154 T1619 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 629154 T1619 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 629155 T1619 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 629155 T1619 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 629155 T1619 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 629155 T1619 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 629156 T1619 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 629156 T1619 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 629156 T1619 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 629157 T1683 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 629157 T1643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 629156 T1665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 629156 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 629160 T1619 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 629161 T1619 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 629161 T1619 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 629161 T1619 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52473/solr
   [junit4]   2> 629162 T1619 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 629162 T1619 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 629164 T1699 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f11928 name:ZooKeeperConnection Watcher:127.0.0.1:52473 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 629164 T1619 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 629166 T1619 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 629167 T1701 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12fff98 name:ZooKeeperConnection Watcher:127.0.0.1:52473/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 629168 T1619 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 629172 T1619 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 629201 T1688 oasc.ZkController.register We are https://127.0.0.1:36580/collection1/ and leader is https://127.0.0.1:36580/collection1/
   [junit4]   2> 629202 T1688 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:36580
   [junit4]   2> 629202 T1688 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 629202 T1688 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 629202 T1688 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 629203 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 629204 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 629204 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 629204 T1688 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 629205 T1644 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 629206 T1644 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:36580",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:36580_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 629208 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 629310 T1665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 629310 T1683 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 629310 T1701 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 629310 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 629310 T1643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 630174 T1619 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45441_
   [junit4]   2> 630176 T1619 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45441_
   [junit4]   2> 630178 T1652 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 630178 T1683 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 630178 T1643 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 630178 T1701 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 630178 T1665 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 630184 T1702 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 630184 T1702 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 630185 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 630185 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 630186 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 630185 T1702 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 630187 T1644 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 630188 T1644 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:45441",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:45441_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 630188 T1644 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 630188 T1644 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 630192 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 630294 T1683 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 630294 T1665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 630294 T1643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 630294 T1652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 630295 T1701 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 631186 T1702 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 631186 T1702 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392721583629/collection1
   [junit4]   2> 631187 T1702 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 631187 T1702 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 631187 T1702 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 631189 T1702 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 631189 T1702 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392721583629/collection1/'
   [junit4]   2> 631189 T1702 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392721583629/collection1/lib/classes/' to classloader
   [junit4]   2> 631189 T1702 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392721583629/collection1/lib/README' to classloader
   [junit4]   2> 631218 T1702 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 631251 T1702 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 631252 T1702 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 631257 T1702 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 631535 T1702 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 631538 T1702 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 631539 T1702 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 631542 T1702 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 631563 T1702 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 631563 T1702 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392721583629/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty3/
   [junit4]   2> 631563 T1702 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@121387e
   [junit4]   2> 631564 T1702 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty3
   [junit4]   2> 631564 T1702 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty3/index/
   [junit4]   2> 631565 T1702 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty3/index' doesn't exist. Creating new index...
   [junit4]   2> 631565 T1702 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty3/index
   [junit4]   2> 631566 T1702 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=18.9658203125, floorSegmentMB=2.013671875, forceMergeDeletesPctAllowed=28.875788144137676, segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3897341227826966
   [junit4]   2> 631568 T1702 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty3/index,segFN=segments_1,generation=1}
   [junit4]   2> 631568 T1702 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 631570 T1702 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 631570 T1702 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 631571 T1702 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 631571 T1702 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 631571 T1702 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 631571 T1702 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 631572 T1702 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 631573 T1702 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 631573 T1702 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 631574 T1702 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 631574 T1702 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 631574 T1702 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 631575 T1702 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 631575 T1702 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 631575 T1702 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 631576 T1702 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 631578 T1702 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 631580 T1702 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 631580 T1702 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 631581 T1702 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1000120485, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 631582 T1702 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392721576617/jetty3/index,segFN=segments_1,generation=1}
   [junit4]   2> 631582 T1702 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 631589 T1702 oass.SolrIndexSearcher.<init> Opening Searcher@37b87c[collection1] main
   [junit4]   2> 631591 T1703 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@37b87c[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 631593 T1702 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 631594 T1619 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 631594 T1619 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 631594 T1706 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:45441 collection:collection1 shard:shard2
   [junit4]   2> 631599 T1706 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 631603 T1706 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 631605 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 631605 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 631605 T1643 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 631605 T1706 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 631606 T1706 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C595 name=collection1 org.apache.solr.core.Solr

[...truncated too long message...]

olExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	  79) Thread[id=2089, name=searcherExecutor-1575-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	  80) Thread[id=2076, name=qtp32917153-2076 Selector0, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
   [junit4]   2> 	        at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)
   [junit4]   2> 	        at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:193)
   [junit4]   2> 	        at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:210)
   [junit4]   2> 	        at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:216)
   [junit4]   2> 	        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:78)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:106)
   [junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:536)
   [junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	  81) Thread[id=2028, name=qtp7406006-2028 Selector1, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
   [junit4]   2> 	        at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)
   [junit4]   2> 	        at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:193)
   [junit4]   2> 	        at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:210)
   [junit4]   2> 	        at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:216)
   [junit4]   2> 	        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:78)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:106)
   [junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:536)
   [junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	  82) Thread[id=2092, name=coreZkRegister-1571-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 7848092 T2060 oaz.ClientCnxn$SendThread.run WARN Session 0x14444acc2ab0029 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> NOTE: test params are: codec=Lucene46: {rnd_b=PostingsFormat(name=NestedPulsing), id=PostingsFormat(name=NestedPulsing), a_si=PostingsFormat(name=TestBloomFilteredLucene41Postings), a_t=PostingsFormat(name=Memory doPackFST= true), _version_=PostingsFormat(name=Memory doPackFST= true)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=pt, timezone=Asia/Kuching
   [junit4]   2> NOTE: Linux 3.8.0-35-generic i386/Oracle Corporation 1.7.0_51 (32-bit)/cpus=8,threads=97,free=213773560,total=485228544
   [junit4]   2> NOTE: All tests run in this JVM: [HdfsLockFactoryTest, SuggesterWFSTTest, TestFastOutputStream, TestCollationFieldDocValues, TestRTGBase, SpellCheckCollatorTest, DefaultValueUpdateProcessorTest, AssignTest, UniqFieldsUpdateProcessorFactoryTest, TestXIncludeConfig, XmlUpdateRequestHandlerTest, BadCopyFieldTest, TestUniqueKeyFieldResource, ConnectionManagerTest, HdfsSyncSliceTest, TestShardHandlerFactory, TestAnalyzedSuggestions, SpellingQueryConverterTest, TestReloadAndDeleteDocs, TestMultiCoreConfBootstrap, TestDefaultSimilarityFactory, IndexBasedSpellCheckerTest, TestFieldResource, TestDFRSimilarityFactory, TestClassNameShortening, TestRecovery, TestSimpleQParserPlugin, TestStressLucene, HdfsUnloadDistributedZkTest, FileUtilsTest, TestLazyCores, SortByFunctionTest, HighlighterTest, BadIndexSchemaTest, CachingDirectoryFactoryTest, OutputWriterTest, CursorPagingTest, HardAutoCommitTest, IndexSchemaRuntimeFieldTest, TestFieldTypeResource, SolrPluginUtilsTest, NotRequiredUniqueKeyTest, TimeZoneUtilsTest, PreAnalyzedUpdateProcessorTest, TestSchemaResource, TestRandomMergePolicy, TestCodecSupport, TestQuerySenderNoQuery, TestSolrXmlPersistence, DirectSolrSpellCheckerTest, DateFieldTest, TestStressRecovery, OpenCloseCoreStressTest, HdfsRecoveryZkTest, HighlighterConfigTest, TestPartialUpdateDeduplication, TestOmitPositions, TestSolrQueryParserDefaultOperatorResource, TestDistributedMissingSort, SuggesterTSTTest, AlternateDirectoryTest, PreAnalyzedFieldTest, TestWordDelimiterFilterFactory, BasicDistributedZk2Test, CoreContainerCoreInitFailuresTest, MultiTermTest, PrimUtilsTest, TestCursorMarkWithoutUniqueKey, RequiredFieldsTest, SpellCheckComponentTest, TestAddFieldRealTimeGet, TestReload, LeaderElectionIntegrationTest, TestImplicitCoreProperties, TestComponentsName, NumericFieldsTest, TestCSVResponseWriter, ZkSolrClientTest, TestRecoveryHdfs, CurrencyFieldOpenExchangeTest, TestCloudManagedSchema, OpenExchangeRatesOrgProviderTest, SolrCoreCheckLockOnStartupTest, TestCollapseQParserPlugin, TestSolrXml, BasicZkTest, ChaosMonkeySafeLeaderTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=B275C9688947A547 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=pt -Dtests.timezone=Asia/Kuching -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | ChaosMonkeySafeLeaderTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([B275C9688947A547]:0)
   [junit4] Completed on J1 in 7226.10s, 1 test, 2 errors <<< FAILURES!

[...truncated 11 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:459: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:439: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1308: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:941: There were test failures: 362 suites, 1563 tests, 1 suite-level error, 1 error, 35 ignored (7 assumptions)

Total time: 155 minutes 46 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_51 -server -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure