You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/02/11 00:44:03 UTC
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_51) - Build #
9432 - Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9432/
Java: 32bit/jdk1.7.0_51 -client -XX:+UseG1GC
2 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
Error Message:
Suite timeout exceeded (>= 7200000 msec).
Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
at __randomizedtesting.SeedInfo.seed([AFCFABA8D1CBAD97]:0)
REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
Test abandoned because suite timeout was reached.
Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([AFCFABA8D1CBAD97]:0)
Build Log:
[...truncated 11238 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4] 2> 587741 T2110 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 587745 T2110 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-ChaosMonkeySafeLeaderTest-1392068590266
[junit4] 2> 587745 T2110 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 587746 T2111 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 587846 T2110 oasc.ZkTestServer.run start zk server on port:45408
[junit4] 2> 587848 T2110 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 587912 T2117 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@edce67 name:ZooKeeperConnection Watcher:127.0.0.1:45408 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 587913 T2110 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 587913 T2110 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 587917 T2110 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 587918 T2119 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@df6db8 name:ZooKeeperConnection Watcher:127.0.0.1:45408/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 587919 T2110 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 587919 T2110 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 587922 T2110 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 587926 T2110 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 587930 T2110 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 587933 T2110 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 587933 T2110 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 587937 T2110 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 587938 T2110 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 587940 T2110 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 587941 T2110 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 587943 T2110 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 587943 T2110 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 587945 T2110 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 587946 T2110 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 587948 T2110 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 587948 T2110 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 587951 T2110 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 587951 T2110 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 587954 T2110 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 587955 T2110 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 587957 T2110 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 587958 T2110 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 587960 T2110 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 587961 T2110 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 587963 T2110 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 587963 T2110 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 588038 T2110 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 588041 T2110 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44353
[junit4] 2> 588041 T2110 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 588042 T2110 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 588042 T2110 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392068590487
[junit4] 2> 588042 T2110 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392068590487/'
[junit4] 2> 588056 T2110 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392068590487/solr.xml
[junit4] 2> 588127 T2110 oasc.CoreContainer.<init> New CoreContainer 31805964
[junit4] 2> 588127 T2110 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392068590487/]
[junit4] 2> 588129 T2110 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 588129 T2110 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 588129 T2110 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 588130 T2110 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 588130 T2110 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 588130 T2110 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 588131 T2110 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 588131 T2110 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 588131 T2110 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 588134 T2110 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 588134 T2110 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 588135 T2110 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 588135 T2110 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:45408/solr
[junit4] 2> 588135 T2110 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 588136 T2110 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 588138 T2130 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6db3a0 name:ZooKeeperConnection Watcher:127.0.0.1:45408 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 588139 T2110 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 588140 T2110 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 588141 T2132 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@897feb name:ZooKeeperConnection Watcher:127.0.0.1:45408/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 588142 T2110 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 588143 T2110 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 588146 T2110 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 588148 T2110 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 588150 T2110 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44353_
[junit4] 2> 588151 T2110 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44353_
[junit4] 2> 588153 T2110 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 588185 T2110 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 588189 T2110 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:44353_
[junit4] 2> 588189 T2110 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 588191 T2110 oasc.Overseer.start Overseer (id=91230607131869187-127.0.0.1:44353_-n_0000000000) starting
[junit4] 2> 588194 T2110 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 588197 T2134 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 588198 T2110 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 588198 T2134 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
[junit4] 2> 588200 T2110 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 588202 T2110 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 588205 T2133 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 588210 T2135 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 588210 T2135 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 588213 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 588213 T2135 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 588215 T2133 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 588215 T2133 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:44353",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:44353_",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 588216 T2133 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 588216 T2133 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 588218 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 588219 T2132 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 589214 T2135 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 589215 T2135 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392068590487/collection1
[junit4] 2> 589215 T2135 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 589216 T2135 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 589216 T2135 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 589217 T2135 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 589218 T2135 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392068590487/collection1/'
[junit4] 2> 589218 T2135 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392068590487/collection1/lib/classes/' to classloader
[junit4] 2> 589219 T2135 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392068590487/collection1/lib/README' to classloader
[junit4] 2> 589279 T2135 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 589326 T2135 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 589328 T2135 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 589336 T2135 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 589734 T2135 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 589737 T2135 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 589739 T2135 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 589742 T2135 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 589759 T2135 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 589759 T2135 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392068590487/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392068590266/control/data/
[junit4] 2> 589759 T2135 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@185c569
[junit4] 2> 589760 T2135 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392068590266/control/data
[junit4] 2> 589760 T2135 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392068590266/control/data/index/
[junit4] 2> 589760 T2135 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392068590266/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 589761 T2135 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392068590266/control/data/index
[junit4] 2> 589762 T2135 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=12, maxMergeAtOnceExplicit=12, maxMergedSegmentMB=73.7197265625, floorSegmentMB=1.755859375, forceMergeDeletesPctAllowed=4.794795395397983, segmentsPerTier=31.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 589765 T2135 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392068590266/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 589765 T2135 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 589768 T2135 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 589768 T2135 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 589769 T2135 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 589769 T2135 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 589769 T2135 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 589769 T2135 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 589770 T2135 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 589770 T2135 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 589770 T2135 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 589771 T2135 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 589771 T2135 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 589771 T2135 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 589772 T2135 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 589772 T2135 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 589772 T2135 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 589773 T2135 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 589773 T2135 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 589776 T2135 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 589778 T2135 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 589779 T2135 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 589779 T2135 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=44, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8483908884472238]
[junit4] 2> 589780 T2135 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392068590266/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 589781 T2135 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 589781 T2135 oass.SolrIndexSearcher.<init> Opening Searcher@3904dd[collection1] main
[junit4] 2> 589784 T2136 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3904dd[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 589786 T2135 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 589787 T2110 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4] 2> 589787 T2110 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 589787 T2139 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:44353 collection:control_collection shard:shard1
[junit4] 2> 589789 T2110 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 589789 T2139 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 589790 T2141 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e7fbd name:ZooKeeperConnection Watcher:127.0.0.1:45408/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 589790 T2110 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 589792 T2110 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 589794 T2110 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 589800 T2139 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 589802 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 589802 T2139 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 589803 T2139 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C652 name=collection1 org.apache.solr.core.SolrCore@14fb945 url=http://127.0.0.1:44353/collection1 node=127.0.0.1:44353_ C652_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:44353, core=collection1, node_name=127.0.0.1:44353_}
[junit4] 2> 589803 T2139 C652 P44353 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:44353/collection1/
[junit4] 2> 589803 T2139 C652 P44353 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 589804 T2133 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 589804 T2139 C652 P44353 oasc.SyncStrategy.syncToMe http://127.0.0.1:44353/collection1/ has no replicas
[junit4] 2> 589804 T2139 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:44353/collection1/ shard1
[junit4] 2> 589804 T2139 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 589806 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 589807 T2132 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 589811 T2133 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 589811 T2141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 589814 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 589883 T2110 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 589884 T2110 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 589887 T2110 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40760
[junit4] 2> 589888 T2110 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 589888 T2110 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 589888 T2110 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392068592316
[junit4] 2> 589889 T2110 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392068592316/'
[junit4] 2> 589902 T2110 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392068592316/solr.xml
[junit4] 2> 589916 T2132 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 589916 T2141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 589949 T2110 oasc.CoreContainer.<init> New CoreContainer 29888630
[junit4] 2> 589950 T2110 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392068592316/]
[junit4] 2> 589951 T2110 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 589951 T2110 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 589952 T2110 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 589952 T2110 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 589952 T2110 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 589953 T2110 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 589953 T2110 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 589953 T2110 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 589954 T2110 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 589956 T2110 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 589956 T2110 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 589957 T2110 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 589957 T2110 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:45408/solr
[junit4] 2> 589958 T2110 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 589958 T2110 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 589960 T2152 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b0e20d name:ZooKeeperConnection Watcher:127.0.0.1:45408 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 589960 T2110 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 589960 T2139 oasc.ZkController.register We are http://127.0.0.1:44353/collection1/ and leader is http://127.0.0.1:44353/collection1/
[junit4] 2> 589961 T2139 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:44353
[junit4] 2> 589961 T2139 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 589961 T2139 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 589962 T2139 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 589963 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 589963 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 589964 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 589964 T2110 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 589964 T2154 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d06636 name:ZooKeeperConnection Watcher:127.0.0.1:45408/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 589964 T2139 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 589965 T2110 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 589966 T2133 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 589967 T2133 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:44353",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:44353_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 589969 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 589970 T2110 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 590071 T2132 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 590071 T2154 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 590071 T2141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 590973 T2110 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40760_
[junit4] 2> 590974 T2110 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40760_
[junit4] 2> 590976 T2141 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 590976 T2154 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 590977 T2132 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 590984 T2155 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 590985 T2155 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 590986 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 590986 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 590986 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 590986 T2155 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 590988 T2133 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 590989 T2133 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:40760",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:40760_",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 590989 T2133 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 590989 T2133 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 590992 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 591096 T2132 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 591096 T2154 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 591096 T2141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 591987 T2155 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 591987 T2155 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392068592316/collection1
[junit4] 2> 591988 T2155 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 591988 T2155 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 591989 T2155 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 591990 T2155 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 591990 T2155 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392068592316/collection1/'
[junit4] 2> 591991 T2155 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392068592316/collection1/lib/classes/' to classloader
[junit4] 2> 591991 T2155 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392068592316/collection1/lib/README' to classloader
[junit4] 2> 592056 T2155 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 592101 T2155 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 592104 T2155 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 592112 T2155 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 592519 T2155 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 592522 T2155 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 592524 T2155 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 592527 T2155 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 592544 T2155 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 592544 T2155 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392068592316/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392068590266/jetty1/
[junit4] 2> 592544 T2155 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@185c569
[junit4] 2> 592545 T2155 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392068590266/jetty1
[junit4] 2> 592545 T2155 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392068590266/jetty1/index/
[junit4] 2> 592546 T2155 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392068590266/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 592546 T2155 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392068590266/jetty1/index
[junit4] 2> 592547 T2155 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=12, maxMergeAtOnceExplicit=12, maxMergedSegmentMB=73.7197265625, floorSegmentMB=1.755859375, forceMergeDeletesPctAllowed=4.794795395397983, segmentsPerTier=31.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 592549 T2155 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392068590266/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 592550 T2155 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 592553 T2155 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 592554 T2155 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 592554 T2155 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 592554 T2155 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 592554 T2155 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 592555 T2155 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 592555 T2155 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 592555 T2155 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 592556 T2155 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 592556 T2155 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 592557 T2155 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 592557 T2155 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 592558 T2155 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 592558 T2155 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 592559 T2155 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 592559 T2155 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 592560 T2155 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 592562 T2155 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 592565 T2155 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 592565 T2155 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 592566 T2155 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=44, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8483908884472238]
[junit4] 2> 592567 T2155 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392068590266/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 592568 T2155 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 592568 T2155 oass.SolrIndexSearcher.<init> Opening Searcher@1b750ed[collection1] main
[junit4] 2> 592573 T2156 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1b750ed[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 592577 T2155 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 592578 T2110 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4] 2> 592578 T2110 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 592578 T2159 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40760 collection:collection1 shard:shard1
[junit4] 2> 592580 T2159 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 592586 T2159 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 592588 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 592588 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 592589 T2159 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 592589 T2159 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 592589 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> ASYNC NEW_CORE C653 name=collection1 org.apache.solr.core.SolrCore@6cdacd url=http://127.0.0.1:40760/collection1 node=127.0.0.1:40760_ C653_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:40760, core=collection1, node_name=127.0.0.1:40760_}
[junit4] 2> 592589 T2159 C653 P40760 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:40760/collection1/
[junit4] 2> 592590 T2133 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 592590 T2159 C653 P40760 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 592590 T2159 C653 P40760 oasc.SyncStrategy.syncToMe http://127.0.0.1:40760/collection1/ has no replicas
[junit4] 2> 592591 T2159 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:40760/collection1/ shard1
[junit4] 2> 592591 T2159 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 592592 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 592597 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 592599 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 592600 T2132 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 592600 T2154 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 592600 T2141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 592647 T2159 oasc.ZkController.register We are http://127.0.0.1:40760/collection1/ and leader is http://127.0.0.1:40760/collection1/
[junit4] 2> 592648 T2159 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40760
[junit4] 2> 592648 T2159 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 592648 T2159 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 592648 T2159 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 592649 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 592650 T2159 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 592651 T2133 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 592652 T2133 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:40760",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:40760_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 592654 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 592655 T2110 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 592655 T2110 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 592657 T2110 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:43194
[junit4] 2> 592658 T2110 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 592658 T2110 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 592659 T2110 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392068595100
[junit4] 2> 592659 T2110 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392068595100/'
[junit4] 2> 592672 T2110 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392068595100/solr.xml
[junit4] 2> 592717 T2110 oasc.CoreContainer.<init> New CoreContainer 16950869
[junit4] 2> 592717 T2110 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392068595100/]
[junit4] 2> 592719 T2110 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 592719 T2110 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 592719 T2110 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 592720 T2110 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 592720 T2110 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 592720 T2110 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 592721 T2110 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 592721 T2110 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 592721 T2110 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 592723 T2110 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 592724 T2110 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 592724 T2110 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 592725 T2110 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:45408/solr
[junit4] 2> 592725 T2110 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 592726 T2110 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 592727 T2170 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f637cb name:ZooKeeperConnection Watcher:127.0.0.1:45408 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 592727 T2110 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 592729 T2110 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 592730 T2172 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9d1fc8 name:ZooKeeperConnection Watcher:127.0.0.1:45408/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 592730 T2110 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 592734 T2110 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 592756 T2141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 592756 T2154 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 592756 T2172 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 592756 T2132 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 593737 T2110 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43194_
[junit4] 2> 593740 T2110 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43194_
[junit4] 2> 593742 T2132 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 593743 T2172 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 593742 T2154 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 593742 T2141 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 593749 T2173 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 593749 T2173 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 593750 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 593751 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 593751 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 593750 T2173 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 593752 T2133 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 593753 T2133 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:43194",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:43194_",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 593753 T2133 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 593754 T2133 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 593758 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 593860 T2132 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 593860 T2154 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 593860 T2141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 593860 T2172 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 594751 T2173 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 594752 T2173 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392068595100/collection1
[junit4] 2> 594752 T2173 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 594753 T2173 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 594753 T2173 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 594754 T2173 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 594755 T2173 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392068595100/collection1/'
[junit4] 2> 594756 T2173 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392068595100/collection1/lib/classes/' to classloader
[junit4] 2> 594756 T2173 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392068595100/collection1/lib/README' to classloader
[junit4] 2> 594836 T2173 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 594882 T2173 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 594884 T2173 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 594892 T2173 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 595300 T2173 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 595303 T2173 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 595305 T2173 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 595308 T2173 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 595330 T2173 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 595331 T2173 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392068595100/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392068590266/jetty2/
[junit4] 2> 595331 T2173 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@185c569
[junit4] 2> 595332 T2173 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392068590266/jetty2
[junit4] 2> 595332 T2173 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392068590266/jetty2/index/
[junit4] 2> 595332 T2173 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392068590266/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 595333 T2173 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392068590266/jetty2/index
[junit4] 2> 595334 T2173 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=12, maxMergeAtOnceExplicit=12, maxMergedSegmentMB=73.7197265625, floorSegmentMB=1.755859375, forceMergeDeletesPctAllowed=4.794795395397983, segmentsPerTier=31.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 595336 T2173 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392068590266/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 595336 T2173 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 595340 T2173 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 595340 T2173 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 595340 T2173 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 595341 T2173 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 595341 T2173 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 595341 T2173 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 595342 T2173 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 595342 T2173 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 595342 T2173 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 595343 T2173 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 595343 T2173 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 595344 T2173 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 595344 T2173 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 595345 T2173 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 595345 T2173 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 595346 T2173 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 595346 T2173 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 595349 T2173 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 595351 T2173 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 595352 T2173 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 595353 T2173 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=44, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8483908884472238]
[junit4] 2> 595354 T2173 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392068590266/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 595354 T2173 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 595354 T2173 oass.SolrIndexSearcher.<init> Opening Searcher@146b836[collection1] main
[junit4] 2> 595359 T2174 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@146b836[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 595363 T2173 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 595363 T2110 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4] 2> 595364 T2110 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 595363 T2177 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:43194 collection:collection1 shard:shard2
[junit4] 2> 595365 T2177 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 595371 T2177 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 595373 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 595373 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 595374 T2177 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 595374 T2177 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 595374 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> ASYNC NEW_CORE C654 name=collection1 org.apache.solr.core.SolrCore@18e0893 url=http://127.0.0.1:43194/collection1 node=127.0.0.1:43194_ C654_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:43194, core=collection1, node_name=127.0.0.1:43194_}
[junit4] 2> 595374 T2177 C654 P43194 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:43194/collection1/
[junit4] 2> 595375 T2177 C654 P43194 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 595376 T2177 C654 P43194 oasc.SyncStrategy.syncToMe http://127.0.0.1:43194/collection1/ has no replicas
[junit4] 2> 595376 T2177 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:43194/collection1/ shard2
[junit4] 2> 595376 T2133 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 595376 T2177 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 595380 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 595384 T2141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 595384 T2132 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 595384 T2154 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 595384 T2172 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 595389 T2133 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 595392 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 595441 T2110 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
[junit4] 2> 595442 T2110 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 595444 T2110 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:46857
[junit4] 2> 595445 T2110 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 595446 T2110 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 595446 T2110 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392068597885
[junit4] 2> 595446 T2110 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392068597885/'
[junit4] 2> 595460 T2110 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392068597885/solr.xml
[junit4] 2> 595495 T2172 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 595495 T2132 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 595495 T2154 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 595495 T2141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 595507 T2110 oasc.CoreContainer.<init> New CoreContainer 17546012
[junit4] 2> 595507 T2110 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392068597885/]
[junit4] 2> 595508 T2110 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 595509 T2110 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 595509 T2110 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 595509 T2110 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 595510 T2110 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 595510 T2110 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 595511 T2110 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 595511 T2110 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 595511 T2110 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 595513 T2110 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 595514 T2110 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 595514 T2110 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 595515 T2110 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:45408/solr
[junit4] 2> 595515 T2110 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 595516 T2110 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 595517 T2188 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@129622b name:ZooKeeperConnection Watcher:127.0.0.1:45408 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 595518 T2110 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 595519 T2110 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 595520 T2190 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@171ca01 name:ZooKeeperConnection Watcher:127.0.0.1:45408/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 595521 T2110 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 595524 T2110 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 595537 T2177 oasc.ZkController.register We are http://127.0.0.1:43194/collection1/ and leader is http://127.0.0.1:43194/collection1/
[junit4] 2> 595538 T2177 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:43194
[junit4] 2> 595538 T2177 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 595538 T2177 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 595538 T2177 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 595539 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 595540 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 595540 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 595540 T2177 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 595541 T2133 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 595541 T2133 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:43194",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:43194_",
[junit4] 2> "shard":"shard2",
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 595543 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 595646 T2141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 595646 T2154 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 595646 T2190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 595646 T2172 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 595646 T2132 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 596527 T2110 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46857_
[junit4] 2> 596529 T2110 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46857_
[junit4] 2> 596531 T2141 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 596531 T2190 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 596531 T2172 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 596532 T2154 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 596532 T2132 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 596538 T2191 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 596539 T2191 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 596540 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 596540 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 596540 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 596540 T2191 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 596542 T2133 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 596542 T2133 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:46857",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:46857_",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 596543 T2133 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 596543 T2133 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 596545 T2132 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 596648 T2172 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 596648 T2190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 596648 T2154 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 596648 T2132 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 596648 T2141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 597541 T2191 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 597541 T2191 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392068597885/collection1
[junit4] 2> 597542 T2191 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 597542 T2191 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 597543 T2191 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 597544 T2191 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 597544 T2191 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392068597885/collection1/'
[junit4] 2> 597545 T2191 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392068597885/collection1/lib/classes/' to classloader
[junit4] 2> 597545 T2191 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392068597885/collection1/lib/README' to classloader
[junit4] 2> 597590 T2191 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 597672 T2191 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 597673 T2191 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 597682 T2191 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 598100 T2191 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 598105 T2191 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 598108 T2191 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 598113 T2191 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 598134 T2191 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 598135 T2191 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392068597885/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392068590266/jetty3/
[junit4] 2> 598135 T2191 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@185c569
[junit4] 2> 598136 T2191 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392068590266/jetty3
[junit4] 2> 598137 T2191 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392068590266/jetty3/index/
[junit4] 2> 598137 T2191 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392068590266/jetty3/index' doesn't exist. Creating new index...
[junit4] 2> 598138 T2191 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392068590266/jetty3/index
[junit4] 2> 598139 T2191 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=12, maxMergeAtOnceExplicit=12, maxMergedSegmentMB=73.7197265625, floorSegmentMB=1.755859375, forceMergeDeletesPctAllowed=4.794795395397983, segmentsPerTier=31.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 598141 T2191 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392068590266/jetty3/index,segFN=segments_1,generation=1}
[junit4] 2> 598142 T2191 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 598145 T2191 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 598146 T2191 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 598146 T2191 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 598146 T2191 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 598147 T2191 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 598147 T2191 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 598147 T2191 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 598148 T2191 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 598148 T2191 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 598149 T2191 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 598149 T2191 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 598150 T2191 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 598150 T2191 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 598151 T2191 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 598151 T2191 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 598152 T2191 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 598152 T2191 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 598155 T2191 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 598157 T2191 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 598158 T2191 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 598159 T2191 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=44, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8483908884472238]
[junit4] 2> 598160 T2191 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392068590266/jetty3/index,segFN=segments_1,generation=1}
[junit4] 2> 598160 T2191 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 598160 T2191 oass.SolrIndexSearcher.<init> Opening Searcher@a88b1[collection1] mai
[...truncated too long message...]
] 2> at java.util.TimerThread.run(Timer.java:505)
[junit4] 2> 85) Thread[id=2496, name=Thread-860-SendThread(localhost.localdomain:45408), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4] 2> at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
[junit4] 2> 86) Thread[id=2314, name=HashSessionScavenger-48, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at java.lang.Object.wait(Native Method)
[junit4] 2> at java.util.TimerThread.mainLoop(Timer.java:552)
[junit4] 2> at java.util.TimerThread.run(Timer.java:505)
[junit4] 2> 87) Thread[id=2471, name=qtp7525411-2471 Selector0, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at java.lang.Thread.isInterrupted(Native Method)
[junit4] 2> at java.lang.Thread.isInterrupted(Thread.java:999)
[junit4] 2> at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:215)
[junit4] 2> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:78)
[junit4] 2> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
[junit4] 2> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
[junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
[junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2> 88) Thread[id=2334, name=HashSessionScavenger-49, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at java.lang.Object.wait(Native Method)
[junit4] 2> at java.util.TimerThread.mainLoop(Timer.java:552)
[junit4] 2> at java.util.TimerThread.run(Timer.java:505)
[junit4] 2> 89) Thread[id=2590, name=Thread-860-SendThread(localhost.localdomain:45408), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4] 2> at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
[junit4] 2> 90) Thread[id=2579, name=HashSessionScavenger-58, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at java.lang.Object.wait(Native Method)
[junit4] 2> at java.util.TimerThread.mainLoop(Timer.java:552)
[junit4] 2> at java.util.TimerThread.run(Timer.java:505)
[junit4] 2> 91) Thread[id=2488, name=qtp27371682-2488 Acceptor0 SelectChannelConnector@127.0.0.1:47904, state=BLOCKED, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at org.eclipse.jetty.server.nio.SelectChannelConnector.getConnection(SelectChannelConnector.java:160)
[junit4] 2> at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:934)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2> 92) Thread[id=2218, name=qtp15178064-2218 Acceptor0 SelectChannelConnector@127.0.0.1:56666, state=BLOCKED, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:102)
[junit4] 2> at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2> 93) Thread[id=2581, name=qtp1270558-2581 Selector1, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at java.lang.Thread.isInterrupted(Native Method)
[junit4] 2> at java.lang.Thread.isInterrupted(Thread.java:999)
[junit4] 2> at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:215)
[junit4] 2> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:78)
[junit4] 2> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
[junit4] 2> at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:106)
[junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:536)
[junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2> NOTE: test params are: codec=Lucene40, sim=DefaultSimilarity, locale=fi, timezone=MET
[junit4] 2> NOTE: Linux 3.8.0-35-generic i386/Oracle Corporation 1.7.0_51 (32-bit)/cpus=8,threads=110,free=340723904,total=536870912
[junit4] 2> NOTE: All tests run in this JVM: [QueryElevationComponentTest, BasicDistributedZkTest, SpellPossibilityIteratorTest, TestHighFrequencyDictionaryFactory, TestTrie, TestSerializedLuceneMatchVersion, CacheHeaderTest, SpatialFilterTest, TestFreeTextSuggestions, TestElisionMultitermQuery, SyncSliceTest, ExpressionTest, AssignTest, UUIDFieldTest, TestRandomMergePolicy, StatelessScriptUpdateProcessorFactoryTest, TestDocumentBuilder, LukeRequestHandlerTest, MBeansHandlerTest, TestJmxMonitoredMap, TestConfig, TestWriterPerf, TestQueryUtils, TestNonNRTOpen, TestPerFieldSimilarity, UpdateParamsTest, DeleteReplicaTest, QueryFacetTest, ZkCLITest, NoFacetTest, RecoveryZkTest, ModifyConfFileTest, TestSolrDeletionPolicy1, TestSolrXMLSerializer, SolrXmlInZkTest, LegacyHTMLStripCharFilterTest, SynonymTokenizerTest, TestAnalyzeInfixSuggestions, ReturnFieldsTest, DocumentBuilderTest, PreAnalyzedUpdateProcessorTest, DeleteShardTest, SliceStateUpdateTest, CoreAdminCreateDiscoverTest, SystemInfoHandlerTest, TestJoin, TestDocBasedVersionConstraints, ExternalFileFieldSortTest, ParsingFieldUpdateProcessorsTest, DistributedQueryElevationComponentTest, DocValuesTest, SliceStateTest, QueryParsingTest, LeaderElectionTest, ShowFileRequestHandlerTest, TestRemoteStreaming, TestCoreContainer, QueryResultKeyTest, TestSort, CSVRequestHandlerTest, DocValuesMissingTest, ZkControllerTest, TestOverriddenPrefixQueryForCustomFieldType, TestPHPSerializedResponseWriter, PrimitiveFieldTypeTest, DateMathParserTest, CircularListTest, TestPseudoReturnFields, EchoParamsTest, MigrateRouteKeyTest, TestInfoStreamLogging, TestFaceting, BJQParserTest, TestPostingsSolrHighlighter, OverseerCollectionProcessorTest, TestSuggestSpellingConverter, TestBadConfig, TestLRUCache, InfoHandlerTest, TestSolrIndexConfig, LoggingHandlerTest, BadComponentTest, TestFieldCollectionResource, RegexBoostProcessorTest, TestDocSet, TestRealTimeGet, SolrCoreTest, MoreLikeThisHandlerTest, SearchHandlerTest, ChaosMonkeySafeLeaderTest]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=AFCFABA8D1CBAD97 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=fi -Dtests.timezone=MET -Dtests.file.encoding=US-ASCII
[junit4] ERROR 0.00s J0 | ChaosMonkeySafeLeaderTest (suite) <<<
[junit4] > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
[junit4] > at __randomizedtesting.SeedInfo.seed([AFCFABA8D1CBAD97]:0)
[junit4] Completed on J0 in 7226.16s, 1 test, 2 errors <<< FAILURES!
[...truncated 11 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:453: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:433: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1268: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:901: There were test failures: 371 suites, 1600 tests, 1 suite-level error, 1 error, 35 ignored (7 assumptions)
Total time: 155 minutes 26 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_51 -client -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_60-ea-b04) -
Build # 9433 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9433/
Java: 32bit/jdk1.7.0_60-ea-b04 -server -XX:+UseConcMarkSweepGC
3 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
Error Message:
Suite timeout exceeded (>= 7200000 msec).
Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
at __randomizedtesting.SeedInfo.seed([3A1595C3BDD01926]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
Error Message:
Captured an uncaught exception in thread: Thread[id=4105, name=RecoveryThread, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=4105, name=RecoveryThread, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]
Caused by: java.lang.AssertionError: liveDocs.count()=15 info.docCount=29 info.getDelCount()=19
at __randomizedtesting.SeedInfo.seed([3A1595C3BDD01926]:0)
at org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.readLiveDocs(Lucene40LiveDocsFormat.java:92)
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:103)
at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:131)
at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:183)
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:97)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:380)
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:271)
at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:204)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1445)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1633)
at org.apache.solr.handler.SnapPuller.openNewSearcherAndUpdateCommitPoint(SnapPuller.java:670)
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:490)
at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:322)
at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:155)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:437)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:247)
REGRESSION: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.testDistribSearch
Error Message:
Test abandoned because suite timeout was reached.
Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([3A1595C3BDD01926]:0)
Build Log:
[...truncated 11098 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
[junit4] 2> 1098776 T3828 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
[junit4] 2> 1098777 T3828 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 1098781 T3828 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeyNothingIsSafeTest-1392081705571
[junit4] 2> 1098782 T3828 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 1098789 T3829 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1098889 T3828 oasc.ZkTestServer.run start zk server on port:47430
[junit4] 2> 1098890 T3828 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1098955 T3835 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@101834c name:ZooKeeperConnection Watcher:127.0.0.1:47430 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1098956 T3828 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1098956 T3828 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1098963 T3828 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1098965 T3837 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cd6208 name:ZooKeeperConnection Watcher:127.0.0.1:47430/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1098965 T3828 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1098965 T3828 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 1098967 T3828 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 1098970 T3828 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 1098972 T3828 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 1098974 T3828 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 1098975 T3828 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 1098978 T3828 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 1098979 T3828 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 1098981 T3828 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1098982 T3828 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1098985 T3828 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 1098985 T3828 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 1098988 T3828 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 1098988 T3828 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 1098990 T3828 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 1098990 T3828 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 1098994 T3828 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 1098995 T3828 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 1099003 T3828 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 1099004 T3828 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 1099006 T3828 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1099006 T3828 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1099008 T3828 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 1099008 T3828 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 1099011 T3828 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 1099011 T3828 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 1099014 T3828 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1099015 T3839 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b8f846 name:ZooKeeperConnection Watcher:127.0.0.1:47430/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1099015 T3828 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1099092 T3828 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1099097 T3828 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 1099099 T3828 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:42629
[junit4] 2> 1099101 T3828 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1099101 T3828 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1099102 T3828 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392081705806
[junit4] 2> 1099102 T3828 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392081705806/'
[junit4] 2> 1099122 T3828 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392081705806/solr.xml
[junit4] 2> 1099160 T3828 oasc.CoreContainer.<init> New CoreContainer 32788818
[junit4] 2> 1099161 T3828 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392081705806/]
[junit4] 2> 1099161 T3828 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1099162 T3828 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 1099162 T3828 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1099162 T3828 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1099163 T3828 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1099163 T3828 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1099163 T3828 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1099163 T3828 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1099164 T3828 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1099168 T3828 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1099169 T3828 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1099169 T3828 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1099170 T3828 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47430/solr
[junit4] 2> 1099170 T3828 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1099172 T3828 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1099173 T3850 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1821bfc name:ZooKeeperConnection Watcher:127.0.0.1:47430 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1099173 T3828 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1099175 T3828 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1099185 T3852 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a0e2f0 name:ZooKeeperConnection Watcher:127.0.0.1:47430/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1099185 T3828 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1099187 T3828 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1099190 T3828 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1099194 T3828 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1099196 T3828 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42629_
[junit4] 2> 1099197 T3828 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42629_
[junit4] 2> 1099200 T3828 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 1099202 T3828 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1099207 T3828 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:42629_
[junit4] 2> 1099207 T3828 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1099209 T3828 oasc.Overseer.start Overseer (id=91231466656825348-127.0.0.1:42629_-n_0000000000) starting
[junit4] 2> 1099212 T3828 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1099215 T3854 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1099215 T3828 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1099216 T3854 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
[junit4] 2> 1099218 T3828 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1099230 T3828 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1099232 T3853 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1099237 T3855 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1099237 T3855 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1099238 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1099238 T3855 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1099239 T3853 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1099240 T3853 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:42629",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:42629_",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 1099241 T3853 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 1099241 T3853 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1099245 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1099248 T3852 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1100239 T3855 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1100239 T3855 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392081705806/collection1
[junit4] 2> 1100239 T3855 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 1100240 T3855 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1100240 T3855 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 1100241 T3855 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1100241 T3855 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392081705806/collection1/'
[junit4] 2> 1100242 T3855 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392081705806/collection1/lib/classes/' to classloader
[junit4] 2> 1100242 T3855 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392081705806/collection1/lib/README' to classloader
[junit4] 2> 1100281 T3855 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 1100308 T3855 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1100309 T3855 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1100350 T3855 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1100617 T3855 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1100619 T3855 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1100620 T3855 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1100625 T3855 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1100658 T3855 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 1100658 T3855 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392081705806/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392081705571/control/data/
[junit4] 2> 1100659 T3855 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1007fc0
[junit4] 2> 1100659 T3855 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392081705571/control/data
[junit4] 2> 1100659 T3855 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392081705571/control/data/index/
[junit4] 2> 1100660 T3855 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392081705571/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 1100660 T3855 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392081705571/control/data/index
[junit4] 2> 1100660 T3855 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=13, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 1100666 T3855 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392081705571/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 1100666 T3855 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1100669 T3855 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1100669 T3855 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1100669 T3855 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1100670 T3855 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1100670 T3855 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1100670 T3855 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1100671 T3855 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1100671 T3855 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1100671 T3855 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1100672 T3855 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1100672 T3855 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1100672 T3855 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1100673 T3855 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1100673 T3855 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1100673 T3855 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1100674 T3855 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 1100674 T3855 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 1100677 T3855 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1100678 T3855 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1100679 T3855 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1100679 T3855 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=10.138671875, floorSegmentMB=1.41015625, forceMergeDeletesPctAllowed=20.53095739357361, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.42013102279078496
[junit4] 2> 1100680 T3855 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392081705571/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 1100680 T3855 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1100680 T3855 oass.SolrIndexSearcher.<init> Opening Searcher@1f955b1[collection1] main
[junit4] 2> 1100682 T3856 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1f955b1[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 1100683 T3855 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1100684 T3828 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 1100684 T3828 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1100684 T3859 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:42629 collection:control_collection shard:shard1
[junit4] 2> 1100687 T3828 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1100687 T3859 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 1100688 T3861 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@419671 name:ZooKeeperConnection Watcher:127.0.0.1:47430/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1100689 T3828 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1100690 T3828 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1100693 T3859 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1100693 T3828 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 1100697 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1100697 T3859 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1100697 T3859 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C6317 name=collection1 org.apache.solr.core.SolrCore@ea8814 url=https://127.0.0.1:42629/collection1 node=127.0.0.1:42629_ C6317_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:42629, core=collection1, node_name=127.0.0.1:42629_}
[junit4] 2> 1100697 T3859 C6317 P42629 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:42629/collection1/
[junit4] 2> 1100698 T3859 C6317 P42629 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1100698 T3859 C6317 P42629 oasc.SyncStrategy.syncToMe https://127.0.0.1:42629/collection1/ has no replicas
[junit4] 2> 1100698 T3859 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:42629/collection1/ shard1
[junit4] 2> 1100698 T3859 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 1100700 T3853 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1100703 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1100708 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1100711 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1100764 T3828 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 1100765 T3828 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1100768 T3828 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 1100769 T3828 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:58992
[junit4] 2> 1100770 T3828 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1100771 T3828 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1100772 T3828 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392081707483
[junit4] 2> 1100772 T3828 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392081707483/'
[junit4] 2> 1100783 T3828 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392081707483/solr.xml
[junit4] 2> 1100810 T3828 oasc.CoreContainer.<init> New CoreContainer 7050232
[junit4] 2> 1100811 T3828 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392081707483/]
[junit4] 2> 1100812 T3828 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1100812 T3828 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 1100812 T3828 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1100813 T3828 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1100813 T3828 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1100813 T3828 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1100814 T3828 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1100813 T3852 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1100813 T3861 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1100814 T3828 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1100815 T3828 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1100820 T3828 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1100820 T3828 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1100821 T3828 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1100821 T3828 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47430/solr
[junit4] 2> 1100821 T3828 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1100822 T3828 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1100825 T3872 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@db74c1 name:ZooKeeperConnection Watcher:127.0.0.1:47430 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1100825 T3828 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1100832 T3828 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1100838 T3874 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bd5606 name:ZooKeeperConnection Watcher:127.0.0.1:47430/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1100838 T3828 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1100842 T3828 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1100859 T3859 oasc.ZkController.register We are https://127.0.0.1:42629/collection1/ and leader is https://127.0.0.1:42629/collection1/
[junit4] 2> 1100859 T3859 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:42629
[junit4] 2> 1100859 T3859 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1100859 T3859 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1100860 T3859 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1100861 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1100861 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1100861 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1100861 T3859 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1100863 T3853 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1100864 T3853 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:42629",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:42629_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 1100868 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1100970 T3861 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1100970 T3874 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1100970 T3852 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1101844 T3828 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58992_
[junit4] 2> 1101846 T3828 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58992_
[junit4] 2> 1101848 T3861 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1101848 T3874 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1101848 T3852 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1101853 T3875 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1101853 T3875 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1101854 T3875 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1101854 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1101855 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1101855 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1101856 T3853 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1101857 T3853 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:58992",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:58992_",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 1101857 T3853 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
[junit4] 2> 1101857 T3853 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1101859 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1101961 T3852 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1101961 T3861 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1101962 T3874 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1102855 T3875 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1102855 T3875 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392081707483/collection1
[junit4] 2> 1102855 T3875 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1102856 T3875 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1102856 T3875 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1102858 T3875 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1102858 T3875 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392081707483/collection1/'
[junit4] 2> 1102859 T3875 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392081707483/collection1/lib/classes/' to classloader
[junit4] 2> 1102859 T3875 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392081707483/collection1/lib/README' to classloader
[junit4] 2> 1102887 T3875 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 1102912 T3875 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1102914 T3875 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1102942 T3875 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1103198 T3875 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1103200 T3875 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1103201 T3875 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1103204 T3875 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1103223 T3875 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 1103223 T3875 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392081707483/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392081705571/jetty1/
[junit4] 2> 1103223 T3875 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1007fc0
[junit4] 2> 1103224 T3875 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392081705571/jetty1
[junit4] 2> 1103224 T3875 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392081705571/jetty1/index/
[junit4] 2> 1103224 T3875 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392081705571/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 1103225 T3875 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392081705571/jetty1/index
[junit4] 2> 1103225 T3875 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=13, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 1103230 T3875 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392081705571/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 1103231 T3875 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1103235 T3875 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1103236 T3875 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1103236 T3875 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1103236 T3875 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1103237 T3875 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1103237 T3875 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1103237 T3875 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1103238 T3875 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1103238 T3875 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1103239 T3875 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1103239 T3875 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1103239 T3875 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1103240 T3875 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1103240 T3875 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1103241 T3875 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1103241 T3875 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 1103242 T3875 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 1103245 T3875 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1103248 T3875 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1103248 T3875 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1103249 T3875 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=10.138671875, floorSegmentMB=1.41015625, forceMergeDeletesPctAllowed=20.53095739357361, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.42013102279078496
[junit4] 2> 1103249 T3875 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392081705571/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 1103250 T3875 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1103250 T3875 oass.SolrIndexSearcher.<init> Opening Searcher@1d87b48[collection1] main
[junit4] 2> 1103253 T3876 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1d87b48[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 1103256 T3875 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1103257 T3828 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 1103257 T3828 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1103257 T3879 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:58992 collection:collection1 shard:shard1
[junit4] 2> 1103259 T3879 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1103276 T3879 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1103280 T3879 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1103280 T3879 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C6318 name=collection1 org.apache.solr.core.SolrCore@1681a8b url=https://127.0.0.1:58992/collection1 node=127.0.0.1:58992_ C6318_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:58992, core=collection1, node_name=127.0.0.1:58992_}
[junit4] 2> 1103280 T3879 C6318 P58992 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:58992/collection1/
[junit4] 2> 1103280 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1103280 T3879 C6318 P58992 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1103281 T3879 C6318 P58992 oasc.SyncStrategy.syncToMe https://127.0.0.1:58992/collection1/ has no replicas
[junit4] 2> 1103281 T3879 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:58992/collection1/ shard1
[junit4] 2> 1103281 T3879 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1103280 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1103281 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1103285 T3853 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1103286 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1103290 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1103359 T3828 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 1103360 T3828 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1103372 T3828 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 1103373 T3828 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:35783
[junit4] 2> 1103375 T3828 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1103376 T3828 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1103376 T3828 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392081710048
[junit4] 2> 1103376 T3828 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392081710048/'
[junit4] 2> 1103390 T3828 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392081710048/solr.xml
[junit4] 2> 1103394 T3861 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1103394 T3852 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1103394 T3874 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1103433 T3828 oasc.CoreContainer.<init> New CoreContainer 28017413
[junit4] 2> 1103434 T3828 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392081710048/]
[junit4] 2> 1103435 T3828 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1103435 T3828 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 1103436 T3828 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1103436 T3828 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1103436 T3828 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1103436 T3879 oasc.ZkController.register We are https://127.0.0.1:58992/collection1/ and leader is https://127.0.0.1:58992/collection1/
[junit4] 2> 1103437 T3828 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1103437 T3879 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:58992
[junit4] 2> 1103438 T3879 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1103438 T3879 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1103438 T3879 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1103437 T3828 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1103439 T3828 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1103440 T3828 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1103439 T3879 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1103439 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1103440 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1103441 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1103442 T3853 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1103443 T3853 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:58992",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:58992_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 1103445 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1103445 T3828 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1103447 T3828 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1103447 T3828 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1103448 T3828 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47430/solr
[junit4] 2> 1103448 T3828 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1103460 T3828 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1103461 T3890 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15120c7 name:ZooKeeperConnection Watcher:127.0.0.1:47430 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1103461 T3828 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1103472 T3830 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1441e891a6b0008, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 1103479 T3828 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1103480 T3892 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1919616 name:ZooKeeperConnection Watcher:127.0.0.1:47430/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1103481 T3828 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1103486 T3828 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1103547 T3852 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1103547 T3874 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1103547 T3892 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1103547 T3861 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1104488 T3828 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35783_
[junit4] 2> 1104490 T3828 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35783_
[junit4] 2> 1104492 T3874 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1104492 T3892 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1104492 T3852 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1104492 T3861 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1104497 T3893 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1104498 T3893 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1104499 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1104499 T3893 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1104499 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1104499 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1104500 T3853 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1104500 T3853 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:35783",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:35783_",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 1104501 T3853 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 1104501 T3853 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1104504 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1104608 T3874 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1104608 T3852 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1104608 T3861 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1104608 T3892 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1105499 T3893 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1105499 T3893 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392081710048/collection1
[junit4] 2> 1105500 T3893 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1105500 T3893 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1105500 T3893 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1105501 T3893 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1105501 T3893 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392081710048/collection1/'
[junit4] 2> 1105502 T3893 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392081710048/collection1/lib/classes/' to classloader
[junit4] 2> 1105502 T3893 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392081710048/collection1/lib/README' to classloader
[junit4] 2> 1105533 T3893 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 1105560 T3893 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1105562 T3893 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1105573 T3893 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1105810 T3893 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1105812 T3893 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1105813 T3893 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1105817 T3893 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1105835 T3893 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 1105836 T3893 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392081710048/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392081705571/jetty2/
[junit4] 2> 1105836 T3893 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1007fc0
[junit4] 2> 1105837 T3893 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392081705571/jetty2
[junit4] 2> 1105837 T3893 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392081705571/jetty2/index/
[junit4] 2> 1105837 T3893 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392081705571/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 1105837 T3893 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392081705571/jetty2/index
[junit4] 2> 1105838 T3893 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=13, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 1105841 T3893 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392081705571/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 1105842 T3893 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1105846 T3893 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1105847 T3893 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1105847 T3893 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1105850 T3893 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1105850 T3893 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1105851 T3893 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1105851 T3893 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1105851 T3893 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1105852 T3893 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1105852 T3893 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1105853 T3893 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1105853 T3893 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1105853 T3893 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1105853 T3893 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1105854 T3893 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1105854 T3893 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 1105855 T3893 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 1105858 T3893 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1105860 T3893 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1105861 T3893 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1105861 T3893 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=10.138671875, floorSegmentMB=1.41015625, forceMergeDeletesPctAllowed=20.53095739357361, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.42013102279078496
[junit4] 2> 1105862 T3893 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392081705571/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 1105862 T3893 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1105863 T3893 oass.SolrIndexSearcher.<init> Opening Searcher@86ee73[collection1] main
[junit4] 2> 1105866 T3894 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@86ee73[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 1105868 T3893 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1105869 T3828 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 1105869 T3828 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1105869 T3897 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:35783 collection:collection1 shard:shard2
[junit4] 2> 1105872 T3897 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 1105877 T3897 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 1105879 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1105880 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1105880 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1105881 T3897 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1105881 T3897 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 1105881 T3853 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> ASYNC NEW_CORE C6319 name=collection1 org.apache.solr.core.SolrCore@144599e url=https://127.0.0.1:35783/collection1 node=127.0.0.1:35783_ C6319_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:35783, core=collection1, node_name=127.0.0.1:35783_}
[junit4] 2> 1105882 T3897 C6319 P35783 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:35783/collection1/
[junit4] 2> 1105882 T3897 C6319 P35783 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1105882 T3897 C6319 P35783 oasc.SyncStrategy.syncToMe https://127.0.0.1:35783/collection1/ has no replicas
[junit4] 2> 1105883 T3897 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:35783/collection1/ shard2
[junit4] 2> 1105883 T3897 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 1105884 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1105887 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1105890 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1105942 T3828 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
[junit4] 2> 1105943 T3828 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1105951 T3828 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 1105952 T3828 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:35186
[junit4] 2> 1105954 T3828 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1105955 T3828 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1105955 T3828 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1392081712660
[junit4] 2> 1105955 T3828 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1392081712660/'
[junit4] 2> 1105966 T3828 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1392081712660/solr.xml
[junit4] 2> 1105994 T3892 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1105994 T3828 oasc.CoreContainer.<init> New CoreContainer 20497582
[junit4] 2> 1105994 T3828 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1392081712660/]
[junit4] 2> 1105994 T3861 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1105994 T3874 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1105994 T3852 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1105996 T3828 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1105996 T3828 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 1105996 T3828 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1105997 T3897 oasc.ZkController.register We are https://127.0.0.1:35783/collection1/ and leader is https://127.0.0.1:35783/collection1/
[junit4] 2> 1105997 T3828 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1105997 T3828 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1105998 T3828 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1105998 T3828 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1105997 T3897 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:35783
[junit4] 2> 1105999 T3897 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1105998 T3828 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1105999 T3828 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1105999 T3897 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1106000 T3897 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1106001 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1106001 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1106001 T3897 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1106001 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1106002 T3853 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1106003 T3853 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:35783",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:35783_",
[junit4] 2> "shard":"shard2",
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 1106004 T3828 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1106005 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1106005 T3828 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1106005 T3828 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1106006 T3828 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47430/solr
[junit4] 2> 1106006 T3828 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1106006 T3828 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1106008 T3908 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1dd0f61 name:ZooKeeperConnection Watcher:127.0.0.1:47430 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1106008 T3828 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1106010 T3828 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1106011 T3910 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@741907 name:ZooKeeperConnection Watcher:127.0.0.1:47430/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1106011 T3828 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1106014 T3828 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1106106 T3852 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1106106 T3861 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1106106 T3874 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1106107 T3910 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1106106 T3892 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1107018 T3828 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35186_
[junit4] 2> 1107020 T3828 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35186_
[junit4] 2> 1107022 T3892 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1107022 T3852 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1107022 T3861 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1107023 T3910 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1107024 T3874 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1107030 T3911 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1107030 T3911 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1107031 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1107031 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1107031 T3911 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1107032 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1107033 T3853 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1107034 T3853 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:35186",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:35186_",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 1107034 T3853 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 1107034 T3853 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1107038 T3852 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1107140 T3852 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1107140 T3874 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1107140 T3892 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1107140 T3910 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1107140 T3861 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1108032 T3911 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1108033 T3911 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1392081712660/collection1
[junit4] 2> 1108033 T3911 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1108034 T3911 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1108034 T3911 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1108035 T3911 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1108035 T3911 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1392081712660/collection1/'
[junit4] 2> 1108035 T3911 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1392081712660/collection1/lib/classes/' to classloader
[junit4] 2> 1108036 T3911 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1392081712660/collection1/lib/README' to classloader
[junit4] 2> 1108076 T3911 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 1108101 T3911 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1108103 T3911 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1108117 T3911 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1108381 T3911 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1108383 T3911 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1108385 T3911 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1108389 T3911 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1108414 T3911 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 1108414 T3911 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1392081712660/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392081705571/jetty3/
[junit4] 2> 1108414 T3911 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1007fc0
[junit4] 2> 1108415 T3911 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392081705571/jetty3
[junit4] 2> 1108415 T3911 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392081705571/jetty3/index/
[junit4] 2> 1108415 T3911 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392081705571/jetty3/index' doesn't exist. Creating new index...
[junit4] 2> 1108416 T3911 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build
[...truncated too long message...]
t org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:536)
[junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2> 35) Thread[id=4341, name=qtp17335480-4341 Selector1, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]
[junit4] 2> at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
[junit4] 2> at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)
[junit4] 2> at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:193)
[junit4] 2> at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:210)
[junit4] 2> at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:216)
[junit4] 2> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:78)
[junit4] 2> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
[junit4] 2> at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:106)
[junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:536)
[junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2> NOTE: test params are: codec=Lucene46: {a_si=PostingsFormat(name=SimpleText), a_t=PostingsFormat(name=Memory doPackFST= false), id=Lucene41(blocksize=128), _version_=PostingsFormat(name=Memory doPackFST= false)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=nl_BE, timezone=Africa/Tripoli
[junit4] 2> NOTE: Linux 3.8.0-35-generic i386/Oracle Corporation 1.7.0_60-ea (32-bit)/cpus=8,threads=43,free=179755920,total=518979584
[junit4] 2> NOTE: All tests run in this JVM: [HighlighterTest, HdfsBasicDistributedZk2Test, PingRequestHandlerTest, ChaosMonkeySafeLeaderTest, HdfsBasicDistributedZkTest, TestFieldTypeResource, TestComponentsName, DeleteReplicaTest, TestDistribDocBasedVersion, TestCloudManagedSchemaAddField, TestQuerySenderNoQuery, TestSchemaResource, PathHierarchyTokenizerFactoryTest, DocumentAnalysisRequestHandlerTest, TestCodecSupport, JSONWriterTest, TestWordDelimiterFilterFactory, TestFoldingMultitermQuery, TestNRTOpen, RequiredFieldsTest, ShardRoutingCustomTest, SpellCheckCollatorTest, SolrCmdDistributorTest, IndexSchemaRuntimeFieldTest, SuggesterTSTTest, DirectUpdateHandlerTest, DateFieldTest, LeaderElectionIntegrationTest, TestStandardQParsers, PluginInfoTest, StressHdfsTest, TestRandomFaceting, ZkNodePropsTest, MultiTermTest, TestJmxIntegration, ZkSolrClientTest, TestBinaryField, SpellCheckComponentTest, OpenCloseCoreStressTest, NumericFieldsTest, DirectSolrConnectionTest, TestBlendedInfixSuggestions, TestSolrXml, OpenExchangeRatesOrgProviderTest, TestNoOpRegenerator, TestMaxScoreQueryParser, TestSolrXmlPersistence, TestManagedSchemaFieldResource, MinimalSchemaTest, AlternateDirectoryTest, TestArbitraryIndexDir, HdfsCollectionsAPIDistributedZkTest, TestDFRSimilarityFactory, SuggestComponentTest, TestRecoveryHdfs, AssignTest, TestFastLRUCache, SolrTestCaseJ4Test, HdfsLockFactoryTest, UniqFieldsUpdateProcessorFactoryTest, TestReload, DefaultValueUpdateProcessorTest, TestAnalyzedSuggestions, TestSerializedLuceneMatchVersion, TestIBSimilarityFactory, BlockDirectoryTest, TestQuerySenderListener, SimplePostToolTest, DisMaxRequestHandlerTest, TestCollationFieldDocValues, TestFieldResource, CurrencyFieldOpenExchangeTest, TestSolrJ, FileUtilsTest, TestNonDefinedSimilarityFactory, TestCollationField, FullSolrCloudDistribCmdsTest, TestQueryTypes, TestLazyCores, TestPhraseSuggestions, TestImplicitCoreProperties, TestCursorMarkWithoutUniqueKey, BasicZkTest, TestShardHandlerFactory, TestStressReorder, CursorMarkTest, DistribCursorPagingTest, CollectionsAPIDistributedZkTest, TestZkChroot, TestDistributedGrouping, TestHashPartitioner, ZkControllerTest, TestMultiCoreConfBootstrap, TestStressVersions, TestRangeQuery, SimpleFacetsTest, TestSolr4Spatial, StatsComponentTest, PeerSyncTest, BadIndexSchemaTest, ConvertedLegacyTest, TestFiltering, ShowFileRequestHandlerTest, SolrIndexSplitterTest, SignatureUpdateProcessorFactoryTest, SolrRequestParserTest, TestStressLucene, SpatialFilterTest, TestCSVLoader, NoCacheHeaderTest, WordBreakSolrSpellCheckerTest, TestPseudoReturnFields, FieldMutatingUpdateProcessorTest, DirectUpdateHandlerOptimizeTest, StatelessScriptUpdateProcessorFactoryTest, TestRemoteStreaming, DistanceFunctionTest, DebugComponentTest, IndexBasedSpellCheckerTest, LukeRequestHandlerTest, DirectSolrSpellCheckerTest, TestReversedWildcardFilterFactory, FileBasedSpellCheckerTest, RequestHandlersTest, DocumentBuilderTest, MoreLikeThisHandlerTest, TestSolrQueryParser, SolrPluginUtilsTest, TestCSVResponseWriter, MBeansHandlerTest, TestPartialUpdateDeduplication, TestBinaryResponseWriter, HighlighterConfigTest, UpdateParamsTest, SolrIndexConfigTest, BadComponentTest, TestSolrDeletionPolicy2, OutputWriterTest, TestFuzzyAnalyzedSuggestions, ExternalFileFieldSortTest, TestSolrCoreProperties, NotRequiredUniqueKeyTest, SpellPossibilityIteratorTest, TestCharFilters, TestXIncludeConfig, EchoParamsTest, TestSweetSpotSimilarityFactory, TestPerFieldSimilarity, TestNumberUtils, TestLMDirichletSimilarityFactory, TestDefaultSimilarityFactory, TestFastWriter, TimeZoneUtilsTest, ResourceLoaderTest, TestFastOutputStream, DateMathParserTest, ChaosMonkeyNothingIsSafeTest]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeyNothingIsSafeTest -Dtests.seed=3A1595C3BDD01926 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=nl_BE -Dtests.timezone=Africa/Tripoli -Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 0.00s J1 | ChaosMonkeyNothingIsSafeTest (suite) <<<
[junit4] > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
[junit4] > at __randomizedtesting.SeedInfo.seed([3A1595C3BDD01926]:0)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=4105, name=RecoveryThread, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]
[junit4] > Caused by: java.lang.AssertionError: liveDocs.count()=15 info.docCount=29 info.getDelCount()=19
[junit4] > at __randomizedtesting.SeedInfo.seed([3A1595C3BDD01926]:0)
[junit4] > at org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.readLiveDocs(Lucene40LiveDocsFormat.java:92)
[junit4] > at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:103)
[junit4] > at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:131)
[junit4] > at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:183)
[junit4] > at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:97)
[junit4] > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:380)
[junit4] > at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:271)
[junit4] > at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:204)
[junit4] > at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1445)
[junit4] > at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1633)
[junit4] > at org.apache.solr.handler.SnapPuller.openNewSearcherAndUpdateCommitPoint(SnapPuller.java:670)
[junit4] > at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:490)
[junit4] > at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:322)
[junit4] > at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:155)
[junit4] > at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:437)
[junit4] > at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:247)
[junit4] Completed on J1 in 7263.98s, 1 test, 3 errors <<< FAILURES!
[...truncated 11 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:453: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:433: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1268: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:901: There were test failures: 372 suites, 1603 tests, 2 suite-level errors, 1 error, 35 ignored (7 assumptions)
Total time: 162 minutes 58 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_60-ea-b04 -server -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure