You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/02/07 23:55:35 UTC
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.6.0_38) - Build #
4200 - Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/4200/
Java: 64bit/jdk1.6.0_38 -XX:+UseParallelGC
2 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
Error Message:
Suite timeout exceeded (>= 7200000 msec).
Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
at __randomizedtesting.SeedInfo.seed([20887FE6FE7A246D]:0)
REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
Test abandoned because suite timeout was reached.
Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([20887FE6FE7A246D]:0)
Build Log:
[...truncated 8793 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4:junit4] 2> 1 T184 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /he_faz/hj
[junit4:junit4] 2> 10 T184 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-ChaosMonkeySafeLeaderTest-1360270464410
[junit4:junit4] 2> 11 T184 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 12 T185 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 13 T185 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4] 2> 13 T185 oazs.ZooKeeperServer.setMinSessionTimeout minSessionTimeout set to -1
[junit4:junit4] 2> 13 T185 oazs.ZooKeeperServer.setMaxSessionTimeout maxSessionTimeout set to -1
[junit4:junit4] 2> 13 T185 oazs.NIOServerCnxnFactory.configure binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4] 2> 14 T185 oazsp.FileTxnSnapLog.save Snapshotting: 0x0 to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/zookeeper/server1/data/version-2/snapshot.0
[junit4:junit4] 2> 112 T184 oasc.ZkTestServer.run start zk server on port:35545
[junit4:junit4] 2> 113 T184 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35545 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@43877c42
[junit4:junit4] 2> 114 T184 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 114 T190 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:35545. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 115 T190 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:35545, initiating session
[junit4:junit4] 2> 115 T186 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:52483
[junit4:junit4] 2> 115 T186 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:52483
[junit4:junit4] 2> 116 T188 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4] 2> 185 T188 oazs.ZooKeeperServer.finishSessionInit Established session 0x13cb670159d0000 with negotiated timeout 10000 for client /127.0.0.1:52483
[junit4:junit4] 2> 185 T190 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:35545, sessionid = 0x13cb670159d0000, negotiated timeout = 10000
[junit4:junit4] 2> 186 T191 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@43877c42 name:ZooKeeperConnection Watcher:127.0.0.1:35545 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 186 T184 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 187 T184 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 193 T189 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13cb670159d0000
[junit4:junit4] 2> 194 T191 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 194 T184 oaz.ZooKeeper.close Session: 0x13cb670159d0000 closed
[junit4:junit4] 2> 194 T186 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:52483 which had sessionid 0x13cb670159d0000
[junit4:junit4] 2> 196 T184 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35545/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@34091772
[junit4:junit4] 2> 197 T184 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 198 T192 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:35545. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 199 T192 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:35545, initiating session
[junit4:junit4] 2> 199 T186 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:52484
[junit4:junit4] 2> 200 T186 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:52484
[junit4:junit4] 2> 201 T188 oazs.ZooKeeperServer.finishSessionInit Established session 0x13cb670159d0001 with negotiated timeout 10000 for client /127.0.0.1:52484
[junit4:junit4] 2> 202 T192 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:35545, sessionid = 0x13cb670159d0001, negotiated timeout = 10000
[junit4:junit4] 2> 202 T193 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@34091772 name:ZooKeeperConnection Watcher:127.0.0.1:35545/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 203 T184 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 203 T184 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 210 T184 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 215 T184 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 220 T184 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 226 T184 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 227 T184 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 236 T184 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 238 T184 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 244 T184 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 245 T184 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 251 T184 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 252 T184 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 258 T184 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 259 T184 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 264 T184 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 265 T184 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 271 T184 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 272 T184 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 278 T184 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 279 T184 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 285 T184 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 286 T184 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 292 T189 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13cb670159d0001
[junit4:junit4] 2> 293 T193 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 293 T186 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:52484 which had sessionid 0x13cb670159d0001
[junit4:junit4] 2> 293 T184 oaz.ZooKeeper.close Session: 0x13cb670159d0001 closed
[junit4:junit4] 2> 485 T184 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 581 T184 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40450
[junit4:junit4] 2> 601 T184 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 601 T184 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 602 T184 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1360270464694
[junit4:junit4] 2> 603 T184 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1360270464694/solr.xml
[junit4:junit4] 2> 603 T184 oasc.CoreContainer.<init> New CoreContainer 1137036716
[junit4:junit4] 2> 604 T184 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1360270464694/'
[junit4:junit4] 2> 605 T184 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1360270464694/'
[junit4:junit4] 2> 621 T184 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 60000
[junit4:junit4] 2> 622 T184 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 622 T184 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 623 T184 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 623 T184 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 624 T184 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 624 T184 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 625 T184 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 626 T184 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 626 T184 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=60000&connTimeout=15000&retry=false
[junit4:junit4] 2> 634 T184 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 647 T184 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:35545/solr
[junit4:junit4] 2> 648 T184 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 648 T184 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35545 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@5481be8a
[junit4:junit4] 2> 649 T184 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 650 T204 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:35545. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 651 T204 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:35545, initiating session
[junit4:junit4] 2> 651 T186 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:52485
[junit4:junit4] 2> 652 T186 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:52485
[junit4:junit4] 2> 653 T188 oazs.ZooKeeperServer.finishSessionInit Established session 0x13cb670159d0002 with negotiated timeout 20000 for client /127.0.0.1:52485
[junit4:junit4] 2> 653 T204 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:35545, sessionid = 0x13cb670159d0002, negotiated timeout = 20000
[junit4:junit4] 2> 654 T205 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5481be8a name:ZooKeeperConnection Watcher:127.0.0.1:35545 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 654 T184 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 656 T189 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13cb670159d0002
[junit4:junit4] 2> 657 T186 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:52485 which had sessionid 0x13cb670159d0002
[junit4:junit4] 2> 656 T184 oaz.ZooKeeper.close Session: 0x13cb670159d0002 closed
[junit4:junit4] 2> 657 T205 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 657 T184 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=60000&connTimeout=15000
[junit4:junit4] 2> 663 T184 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35545/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@34b84c44
[junit4:junit4] 2> 664 T184 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 665 T206 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:35545. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 666 T206 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:35545, initiating session
[junit4:junit4] 2> 666 T186 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:52486
[junit4:junit4] 2> 666 T186 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:52486
[junit4:junit4] 2> 667 T188 oazs.ZooKeeperServer.finishSessionInit Established session 0x13cb670159d0003 with negotiated timeout 20000 for client /127.0.0.1:52486
[junit4:junit4] 2> 668 T206 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:35545, sessionid = 0x13cb670159d0003, negotiated timeout = 20000
[junit4:junit4] 2> 669 T207 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@34b84c44 name:ZooKeeperConnection Watcher:127.0.0.1:35545/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 669 T184 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 676 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d0003 type:create cxid:0x2 zxid:0x1a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 677 T184 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 680 T184 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40450_he_faz%2Fhj
[junit4:junit4] 2> 681 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d0003 type:delete cxid:0x7 zxid:0x1c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:40450_he_faz%2Fhj Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:40450_he_faz%2Fhj
[junit4:junit4] 2> 682 T184 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40450_he_faz%2Fhj
[junit4:junit4] 2> 686 T184 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 692 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d0003 type:delete cxid:0x15 zxid:0x21 txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4] 2> 692 T184 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 695 T184 oasc.Overseer.start Overseer (id=89146685155770371-127.0.0.1:40450_he_faz%2Fhj-n_0000000000) starting
[junit4:junit4] 2> 695 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d0003 type:create cxid:0x1a zxid:0x23 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 696 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d0003 type:create cxid:0x1b zxid:0x24 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 697 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d0003 type:create cxid:0x1c zxid:0x25 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 698 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d0003 type:create cxid:0x1d zxid:0x26 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 698 T209 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 699 T184 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 701 T184 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 703 T208 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 706 T210 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1360270464694/collection1
[junit4:junit4] 2> 706 T210 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 707 T210 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 707 T210 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 708 T210 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1360270464694/collection1/'
[junit4:junit4] 2> 709 T210 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1360270464694/collection1/lib/README' to classloader
[junit4:junit4] 2> 709 T210 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1360270464694/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 738 T210 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 777 T210 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 778 T210 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 783 T210 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 1116 T210 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 1125 T210 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 1129 T210 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 1144 T210 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1148 T210 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1151 T210 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 1152 T210 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 1153 T210 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1360270464694/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/control/data/
[junit4:junit4] 2> 1154 T210 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@f5bfdbd
[junit4:junit4] 2> 1155 T210 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 1162 T210 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/control/data forceNew:false
[junit4:junit4] 2> 1162 T210 oasc.CachingDirectoryFactory.close Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/control/data
[junit4:junit4] 2> 1162 T210 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/control/data/index/
[junit4:junit4] 2> 1163 T210 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 1163 T210 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/control/data/index forceNew:false
[junit4:junit4] 2> 1167 T210 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=RateLimitedDirectoryWrapper(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2ebade60),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 1167 T210 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 1167 T210 oasc.CachingDirectoryFactory.close Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/control/data/index
[junit4:junit4] 2> 1168 T210 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4] 2> 1169 T210 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 1170 T210 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4] 2> 1170 T210 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 1170 T210 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4] 2> 1171 T210 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 1171 T210 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 1171 T210 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 1172 T210 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 1172 T210 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 1173 T210 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4] 2> 1173 T210 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4] 2> 1173 T210 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 1174 T210 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4] 2> 1174 T210 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4] 2> 1174 T210 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4] 2> 1175 T210 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 1175 T210 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 1175 T210 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 1176 T210 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 1176 T210 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 1177 T210 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 1177 T210 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 1177 T210 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 1178 T210 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 1178 T210 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4] 2> 1179 T210 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4] 2> 1180 T210 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 1182 T210 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4] 2> 1183 T210 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4] 2> 1185 T210 oasc.CachingDirectoryFactory.close Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/control/data
[junit4:junit4] 2> 1185 T210 oass.SolrIndexSearcher.<init> Opening Searcher@76557c15 main
[junit4:junit4] 2> 1186 T210 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 1186 T210 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 1187 T210 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4] 2> 1190 T210 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4] 2> 1524 T211 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@76557c15 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 1528 T210 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 1528 T210 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1529 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d0003 type:create cxid:0x50 zxid:0x29 txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4] 2> 2209 T208 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2210 T208 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "node_name":"127.0.0.1:40450_he_faz%2Fhj",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40450/he_faz/hj"}
[junit4:junit4] 2> 2211 T208 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 2213 T208 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 2214 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d0003 type:create cxid:0x58 zxid:0x2c txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4] 2> 2224 T207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 2532 T210 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 2532 T210 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40450/he_faz/hj collection:control_collection shard:shard1
[junit4:junit4] 2> 2534 T210 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 2546 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d0003 type:delete cxid:0x72 zxid:0x36 txntype:-1 reqpath:n/a Error Path:/solr/collections/control_collection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/control_collection/leaders
[junit4:junit4] 2> 2547 T210 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 2548 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d0003 type:create cxid:0x73 zxid:0x37 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 2551 T210 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 2552 T210 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 2552 T210 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:40450/he_faz/hj/collection1/
[junit4:junit4] 2> 2553 T210 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 2554 T210 oasc.SyncStrategy.syncToMe http://127.0.0.1:40450/he_faz/hj/collection1/ has no replicas
[junit4:junit4] 2> 2554 T210 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:40450/he_faz/hj/collection1/
[junit4:junit4] 2> 2555 T210 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 2562 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d0003 type:create cxid:0x7d zxid:0x3b txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 3732 T208 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3747 T207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 3768 T210 oasc.ZkController.register We are http://127.0.0.1:40450/he_faz/hj/collection1/ and leader is http://127.0.0.1:40450/he_faz/hj/collection1/
[junit4:junit4] 2> 3768 T210 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40450/he_faz/hj
[junit4:junit4] 2> 3769 T210 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 3769 T210 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 3769 T210 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 3773 T210 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3775 T184 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 3776 T184 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 3798 T184 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 3804 T184 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 3808 T184 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35545/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@6aeeefcf
[junit4:junit4] 2> 3809 T184 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3809 T212 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:35545. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 3810 T212 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:35545, initiating session
[junit4:junit4] 2> 3810 T186 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:52491
[junit4:junit4] 2> 3811 T186 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:52491
[junit4:junit4] 2> 3812 T188 oazs.ZooKeeperServer.finishSessionInit Established session 0x13cb670159d0004 with negotiated timeout 10000 for client /127.0.0.1:52491
[junit4:junit4] 2> 3812 T212 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:35545, sessionid = 0x13cb670159d0004, negotiated timeout = 10000
[junit4:junit4] 2> 3812 T213 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6aeeefcf name:ZooKeeperConnection Watcher:127.0.0.1:35545/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3813 T184 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3815 T184 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 3818 T184 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 3874 T184 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 3876 T184 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:36781
[junit4:junit4] 2> 3877 T184 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 3877 T184 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 3877 T184 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1360270468217
[junit4:junit4] 2> 3878 T184 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1360270468217/solr.xml
[junit4:junit4] 2> 3878 T184 oasc.CoreContainer.<init> New CoreContainer 1809159097
[junit4:junit4] 2> 3878 T184 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1360270468217/'
[junit4:junit4] 2> 3879 T184 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1360270468217/'
[junit4:junit4] 2> 3891 T184 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 60000
[junit4:junit4] 2> 3892 T184 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 3892 T184 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 3892 T184 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 3893 T184 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 3893 T184 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 3893 T184 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 3893 T184 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 3894 T184 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 3894 T184 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=60000&connTimeout=15000&retry=false
[junit4:junit4] 2> 3900 T184 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 3909 T184 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:35545/solr
[junit4:junit4] 2> 3909 T184 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 3910 T184 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35545 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@25d285b
[junit4:junit4] 2> 3911 T184 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3911 T223 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:35545. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 3912 T223 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:35545, initiating session
[junit4:junit4] 2> 3912 T186 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:52492
[junit4:junit4] 2> 3912 T186 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:52492
[junit4:junit4] 2> 3913 T188 oazs.ZooKeeperServer.finishSessionInit Established session 0x13cb670159d0005 with negotiated timeout 20000 for client /127.0.0.1:52492
[junit4:junit4] 2> 3913 T223 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:35545, sessionid = 0x13cb670159d0005, negotiated timeout = 20000
[junit4:junit4] 2> 3914 T224 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@25d285b name:ZooKeeperConnection Watcher:127.0.0.1:35545 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3914 T184 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3915 T189 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13cb670159d0005
[junit4:junit4] 2> 3915 T224 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 3916 T186 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:52492 which had sessionid 0x13cb670159d0005
[junit4:junit4] 2> 3915 T184 oaz.ZooKeeper.close Session: 0x13cb670159d0005 closed
[junit4:junit4] 2> 3916 T184 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=60000&connTimeout=15000
[junit4:junit4] 2> 3922 T184 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35545/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@3a3c6542
[junit4:junit4] 2> 3924 T225 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:35545. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 3923 T184 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3924 T225 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:35545, initiating session
[junit4:junit4] 2> 3925 T186 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:52493
[junit4:junit4] 2> 3925 T186 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:52493
[junit4:junit4] 2> 3926 T188 oazs.ZooKeeperServer.finishSessionInit Established session 0x13cb670159d0006 with negotiated timeout 20000 for client /127.0.0.1:52493
[junit4:junit4] 2> 3926 T225 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:35545, sessionid = 0x13cb670159d0006, negotiated timeout = 20000
[junit4:junit4] 2> 3927 T226 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a3c6542 name:ZooKeeperConnection Watcher:127.0.0.1:35545/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3927 T184 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3928 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d0006 type:create cxid:0x1 zxid:0x49 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 3930 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d0006 type:create cxid:0x2 zxid:0x4a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 3932 T184 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36781_he_faz%2Fhj
[junit4:junit4] 2> 3932 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d0006 type:delete cxid:0x4 zxid:0x4b txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:36781_he_faz%2Fhj Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:36781_he_faz%2Fhj
[junit4:junit4] 2> 3933 T184 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36781_he_faz%2Fhj
[junit4:junit4] 2> 3936 T207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 3937 T213 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 3940 T207 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 3943 T184 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 3951 T227 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1360270468217/collection1
[junit4:junit4] 2> 3952 T227 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 3953 T227 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 3953 T227 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 3956 T227 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1360270468217/collection1/'
[junit4:junit4] 2> 3957 T227 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1360270468217/collection1/lib/README' to classloader
[junit4:junit4] 2> 3957 T227 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1360270468217/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 4006 T227 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 4049 T227 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 4050 T227 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 4054 T227 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 4356 T227 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 4361 T227 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 4364 T227 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 4379 T227 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 4382 T227 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 4385 T227 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 4387 T227 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 4388 T227 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1360270468217/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty1/
[junit4:junit4] 2> 4388 T227 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@f5bfdbd
[junit4:junit4] 2> 4389 T227 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 4389 T227 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty1 forceNew:false
[junit4:junit4] 2> 4390 T227 oasc.CachingDirectoryFactory.close Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty1
[junit4:junit4] 2> 4390 T227 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty1/index/
[junit4:junit4] 2> 4390 T227 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 4391 T227 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty1/index forceNew:false
[junit4:junit4] 2> 4394 T227 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=RateLimitedDirectoryWrapper(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@13d723e6),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 4394 T227 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 4394 T227 oasc.CachingDirectoryFactory.close Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty1/index
[junit4:junit4] 2> 4395 T227 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4] 2> 4396 T227 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 4396 T227 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4] 2> 4397 T227 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 4397 T227 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4] 2> 4397 T227 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 4398 T227 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 4398 T227 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 4398 T227 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 4399 T227 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 4399 T227 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4] 2> 4400 T227 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4] 2> 4400 T227 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 4401 T227 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4] 2> 4401 T227 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4] 2> 4401 T227 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4] 2> 4402 T227 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 4402 T227 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 4402 T227 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 4403 T227 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 4403 T227 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 4404 T227 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 4404 T227 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 4405 T227 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 4405 T227 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 4406 T227 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4] 2> 4406 T227 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4] 2> 4408 T227 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 4409 T227 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4] 2> 4411 T227 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4] 2> 4412 T227 oasc.CachingDirectoryFactory.close Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty1
[junit4:junit4] 2> 4413 T227 oass.SolrIndexSearcher.<init> Opening Searcher@37d40164 main
[junit4:junit4] 2> 4413 T227 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 4413 T227 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 4414 T227 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4] 2> 4418 T227 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4] 2> 4441 T228 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@37d40164 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 4446 T227 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 4446 T227 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 5258 T208 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 5259 T208 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "node_name":"127.0.0.1:40450_he_faz%2Fhj",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40450/he_faz/hj"}
[junit4:junit4] 2> 5266 T208 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:36781_he_faz%2Fhj",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:36781/he_faz/hj"}
[junit4:junit4] 2> 5266 T208 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 3
[junit4:junit4] 2> 5267 T208 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 5274 T213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 5274 T226 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 5274 T207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 5447 T227 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 5448 T227 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:36781/he_faz/hj collection:collection1 shard:shard1
[junit4:junit4] 2> 5449 T227 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 5459 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d0006 type:delete cxid:0x43 zxid:0x5a txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4] 2> 5460 T227 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 5461 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d0006 type:create cxid:0x44 zxid:0x5b txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 5464 T227 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 5464 T227 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 5465 T227 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:36781/he_faz/hj/collection1/
[junit4:junit4] 2> 5465 T227 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 5466 T227 oasc.SyncStrategy.syncToMe http://127.0.0.1:36781/he_faz/hj/collection1/ has no replicas
[junit4:junit4] 2> 5466 T227 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:36781/he_faz/hj/collection1/
[junit4:junit4] 2> 5467 T227 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 5473 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d0006 type:create cxid:0x4e zxid:0x5f txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 6786 T208 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 6804 T213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 6804 T207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 6804 T226 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 6829 T227 oasc.ZkController.register We are http://127.0.0.1:36781/he_faz/hj/collection1/ and leader is http://127.0.0.1:36781/he_faz/hj/collection1/
[junit4:junit4] 2> 6830 T227 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:36781/he_faz/hj
[junit4:junit4] 2> 6830 T227 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 6830 T227 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 6831 T227 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 6835 T227 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 6837 T184 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 6837 T184 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 6838 T184 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 6905 T184 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 6907 T184 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49471
[junit4:junit4] 2> 6908 T184 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 6908 T184 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 6909 T184 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1360270471244
[junit4:junit4] 2> 6909 T184 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1360270471244/solr.xml
[junit4:junit4] 2> 6909 T184 oasc.CoreContainer.<init> New CoreContainer 1886693295
[junit4:junit4] 2> 6910 T184 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1360270471244/'
[junit4:junit4] 2> 6910 T184 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1360270471244/'
[junit4:junit4] 2> 6923 T184 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 60000
[junit4:junit4] 2> 6924 T184 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 6924 T184 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 6924 T184 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 6925 T184 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 6925 T184 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 6925 T184 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 6926 T184 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 6926 T184 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 6926 T184 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=60000&connTimeout=15000&retry=false
[junit4:junit4] 2> 6932 T184 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 6942 T184 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:35545/solr
[junit4:junit4] 2> 6942 T184 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 6943 T184 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35545 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@76e44d85
[junit4:junit4] 2> 6944 T184 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 6944 T238 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:35545. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 6945 T238 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:35545, initiating session
[junit4:junit4] 2> 6945 T186 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:52497
[junit4:junit4] 2> 6945 T186 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:52497
[junit4:junit4] 2> 6946 T188 oazs.ZooKeeperServer.finishSessionInit Established session 0x13cb670159d0007 with negotiated timeout 20000 for client /127.0.0.1:52497
[junit4:junit4] 2> 6946 T238 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:35545, sessionid = 0x13cb670159d0007, negotiated timeout = 20000
[junit4:junit4] 2> 6946 T239 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@76e44d85 name:ZooKeeperConnection Watcher:127.0.0.1:35545 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 6946 T184 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 6948 T189 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13cb670159d0007
[junit4:junit4] 2> 6950 T239 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 6950 T186 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:52497 which had sessionid 0x13cb670159d0007
[junit4:junit4] 2> 6949 T184 oaz.ZooKeeper.close Session: 0x13cb670159d0007 closed
[junit4:junit4] 2> 6950 T184 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=60000&connTimeout=15000
[junit4:junit4] 2> 6953 T184 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35545/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@101575b4
[junit4:junit4] 2> 6954 T184 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 6954 T240 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:35545. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 6955 T240 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:35545, initiating session
[junit4:junit4] 2> 6955 T186 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:52498
[junit4:junit4] 2> 6955 T186 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:52498
[junit4:junit4] 2> 6956 T188 oazs.ZooKeeperServer.finishSessionInit Established session 0x13cb670159d0008 with negotiated timeout 20000 for client /127.0.0.1:52498
[junit4:junit4] 2> 6956 T240 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:35545, sessionid = 0x13cb670159d0008, negotiated timeout = 20000
[junit4:junit4] 2> 6957 T241 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@101575b4 name:ZooKeeperConnection Watcher:127.0.0.1:35545/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 6957 T184 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 6958 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d0008 type:create cxid:0x1 zxid:0x6c txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 6960 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d0008 type:create cxid:0x2 zxid:0x6d txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 6962 T184 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49471_he_faz%2Fhj
[junit4:junit4] 2> 6963 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d0008 type:delete cxid:0x4 zxid:0x6e txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:49471_he_faz%2Fhj Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:49471_he_faz%2Fhj
[junit4:junit4] 2> 6964 T184 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49471_he_faz%2Fhj
[junit4:junit4] 2> 6967 T207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 6968 T226 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 6968 T213 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 6968 T213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 6971 T207 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 6971 T226 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 6975 T184 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 6980 T242 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1360270471244/collection1
[junit4:junit4] 2> 6980 T242 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 6981 T242 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 6981 T242 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 6982 T242 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1360270471244/collection1/'
[junit4:junit4] 2> 6983 T242 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1360270471244/collection1/lib/README' to classloader
[junit4:junit4] 2> 6983 T242 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1360270471244/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 7012 T242 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 7051 T242 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 7052 T242 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 7056 T242 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 7371 T242 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 7376 T242 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 7378 T242 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 7392 T242 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 7395 T242 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 7398 T242 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 7400 T242 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 7401 T242 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1360270471244/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty2/
[junit4:junit4] 2> 7401 T242 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@f5bfdbd
[junit4:junit4] 2> 7402 T242 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 7402 T242 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty2 forceNew:false
[junit4:junit4] 2> 7403 T242 oasc.CachingDirectoryFactory.close Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty2
[junit4:junit4] 2> 7403 T242 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty2/index/
[junit4:junit4] 2> 7403 T242 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 7404 T242 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty2/index forceNew:false
[junit4:junit4] 2> 7409 T242 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=RateLimitedDirectoryWrapper(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@5f450701),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 7409 T242 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 7410 T242 oasc.CachingDirectoryFactory.close Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty2/index
[junit4:junit4] 2> 7411 T242 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4] 2> 7414 T242 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 7414 T242 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4] 2> 7415 T242 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 7415 T242 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4] 2> 7416 T242 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 7417 T242 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 7417 T242 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 7418 T242 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 7419 T242 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 7420 T242 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4] 2> 7421 T242 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4] 2> 7421 T242 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 7421 T242 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4] 2> 7422 T242 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4] 2> 7422 T242 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4] 2> 7422 T242 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 7423 T242 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 7423 T242 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 7424 T242 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 7424 T242 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 7425 T242 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 7425 T242 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 7426 T242 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 7426 T242 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 7427 T242 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4] 2> 7427 T242 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4] 2> 7429 T242 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 7430 T242 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4] 2> 7431 T242 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4] 2> 7433 T242 oasc.CachingDirectoryFactory.close Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty2
[junit4:junit4] 2> 7434 T242 oass.SolrIndexSearcher.<init> Opening Searcher@11e95d95 main
[junit4:junit4] 2> 7434 T242 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 7434 T242 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 7435 T242 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4] 2> 7438 T242 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4] 2> 7464 T243 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@11e95d95 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 7474 T242 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 7474 T242 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 8315 T208 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 8318 T208 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:36781_he_faz%2Fhj",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:36781/he_faz/hj"}
[junit4:junit4] 2> 8325 T208 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:49471_he_faz%2Fhj",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49471/he_faz/hj"}
[junit4:junit4] 2> 8325 T208 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 8326 T208 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4:junit4] 2> 8333 T213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 8334 T226 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 8334 T207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 8333 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 8476 T242 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 8477 T242 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49471/he_faz/hj collection:collection1 shard:shard2
[junit4:junit4] 2> 8479 T242 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 8490 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d0008 type:delete cxid:0x42 zxid:0x7c txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders/shard2 Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders/shard2
[junit4:junit4] 2> 8491 T242 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 8492 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d0008 type:create cxid:0x43 zxid:0x7d txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 8496 T242 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 8496 T242 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 8497 T242 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:49471/he_faz/hj/collection1/
[junit4:junit4] 2> 8497 T242 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 8497 T242 oasc.SyncStrategy.syncToMe http://127.0.0.1:49471/he_faz/hj/collection1/ has no replicas
[junit4:junit4] 2> 8498 T242 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49471/he_faz/hj/collection1/
[junit4:junit4] 2> 8498 T242 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 8506 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d0008 type:create cxid:0x4c zxid:0x80 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 9844 T208 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 9851 T213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9851 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9851 T226 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9851 T207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9862 T242 oasc.ZkController.register We are http://127.0.0.1:49471/he_faz/hj/collection1/ and leader is http://127.0.0.1:49471/he_faz/hj/collection1/
[junit4:junit4] 2> 9863 T242 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49471/he_faz/hj
[junit4:junit4] 2> 9863 T242 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 9863 T242 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 9863 T242 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 9865 T242 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 9867 T184 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 9867 T184 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 9867 T184 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 9925 T184 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 9927 T184 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:38025
[junit4:junit4] 2> 9928 T184 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 9928 T184 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 9929 T184 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1360270474270
[junit4:junit4] 2> 9929 T184 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1360270474270/solr.xml
[junit4:junit4] 2> 9929 T184 oasc.CoreContainer.<init> New CoreContainer 467610514
[junit4:junit4] 2> 9930 T184 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1360270474270/'
[junit4:junit4] 2> 9930 T184 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1360270474270/'
[junit4:junit4] 2> 9944 T184 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 60000
[junit4:junit4] 2> 9944 T184 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 9945 T184 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 9945 T184 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 9945 T184 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 9946 T184 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 9946 T184 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 9946 T184 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 9947 T184 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 9947 T184 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=60000&connTimeout=15000&retry=false
[junit4:junit4] 2> 9954 T184 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 9964 T184 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:35545/solr
[junit4:junit4] 2> 9964 T184 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 9964 T184 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35545 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@5cada3d6
[junit4:junit4] 2> 9965 T184 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 9966 T253 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:35545. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 9966 T253 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:35545, initiating session
[junit4:junit4] 2> 9967 T186 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:52501
[junit4:junit4] 2> 9967 T186 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:52501
[junit4:junit4] 2> 9968 T188 oazs.ZooKeeperServer.finishSessionInit Established session 0x13cb670159d0009 with negotiated timeout 20000 for client /127.0.0.1:52501
[junit4:junit4] 2> 9968 T253 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:35545, sessionid = 0x13cb670159d0009, negotiated timeout = 20000
[junit4:junit4] 2> 9968 T254 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5cada3d6 name:ZooKeeperConnection Watcher:127.0.0.1:35545 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 9968 T184 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 9969 T189 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13cb670159d0009
[junit4:junit4] 2> 9970 T254 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 9970 T186 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:52501 which had sessionid 0x13cb670159d0009
[junit4:junit4] 2> 9970 T184 oaz.ZooKeeper.close Session: 0x13cb670159d0009 closed
[junit4:junit4] 2> 9971 T184 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=60000&connTimeout=15000
[junit4:junit4] 2> 9973 T184 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35545/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@45a2f279
[junit4:junit4] 2> 9974 T184 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 9974 T255 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:35545. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 9975 T255 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:35545, initiating session
[junit4:junit4] 2> 9975 T186 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:52502
[junit4:junit4] 2> 9978 T186 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:52502
[junit4:junit4] 2> 9979 T188 oazs.ZooKeeperServer.finishSessionInit Established session 0x13cb670159d000a with negotiated timeout 20000 for client /127.0.0.1:52502
[junit4:junit4] 2> 9979 T255 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:35545, sessionid = 0x13cb670159d000a, negotiated timeout = 20000
[junit4:junit4] 2> 9980 T256 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@45a2f279 name:ZooKeeperConnection Watcher:127.0.0.1:35545/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 9980 T184 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 9981 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d000a type:create cxid:0x1 zxid:0x8d txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 9982 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d000a type:create cxid:0x2 zxid:0x8e txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 9982 T184 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38025_he_faz%2Fhj
[junit4:junit4] 2> 9983 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d000a type:delete cxid:0x4 zxid:0x8f txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:38025_he_faz%2Fhj Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:38025_he_faz%2Fhj
[junit4:junit4] 2> 9984 T184 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38025_he_faz%2Fhj
[junit4:junit4] 2> 9986 T226 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9986 T207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9987 T213 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 9987 T241 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 9987 T213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 9987 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 9989 T226 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 9989 T207 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 9991 T184 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 9996 T257 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1360270474270/collection1
[junit4:junit4] 2> 9996 T257 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 9997 T257 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 9997 T257 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 9998 T257 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1360270474270/collection1/'
[junit4:junit4] 2> 9999 T257 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1360270474270/collection1/lib/README' to classloader
[junit4:junit4] 2> 9999 T257 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1360270474270/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 10025 T257 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 10061 T257 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 10062 T257 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 10065 T257 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 10363 T257 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 10368 T257 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 10371 T257 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 10384 T257 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 10388 T257 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 10391 T257 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 10393 T257 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 10394 T257 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1360270474270/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty3/
[junit4:junit4] 2> 10394 T257 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@f5bfdbd
[junit4:junit4] 2> 10395 T257 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 10395 T257 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty3 forceNew:false
[junit4:junit4] 2> 10396 T257 oasc.CachingDirectoryFactory.close Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty3
[junit4:junit4] 2> 10396 T257 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty3/index/
[junit4:junit4] 2> 10396 T257 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 10397 T257 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty3/index forceNew:false
[junit4:junit4] 2> 10400 T257 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=RateLimitedDirectoryWrapper(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@3f1a2258),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 10400 T257 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 10400 T257 oasc.CachingDirectoryFactory.close Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty3/index
[junit4:junit4] 2> 10401 T257 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4] 2> 10402 T257 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 10402 T257 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4] 2> 10403 T257 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 10403 T257 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4] 2> 10404 T257 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 10404 T257 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 10404 T257 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 10405 T257 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 10405 T257 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 10406 T257 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4] 2> 10406 T257 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4] 2> 10406 T257 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 10407 T257 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4] 2> 10407 T257 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4] 2> 10407 T257 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4] 2> 10408 T257 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 10409 T257 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 10409 T257 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 10410 T257 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 10410 T257 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 10411 T257 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 10411 T257 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 10412 T257 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 10412 T257 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 10412 T257 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4] 2> 10413 T257 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4] 2> 10415 T257 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 10416 T257 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4] 2> 10418 T257 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4] 2> 10419 T257 oasc.CachingDirectoryFactory.close Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty3
[junit4:junit4] 2> 10420 T257 oass.SolrIndexSearcher.<init> Opening Searcher@46c04fc1 main
[junit4:junit4] 2> 10421 T257 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 10421 T257 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 10421 T257 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4] 2> 10425 T257 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4] 2> 10447 T258 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@46c04fc1 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 10454 T257 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 10454 T257 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 11359 T208 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 11361 T208 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:49471_he_faz%2Fhj",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49471/he_faz/hj"}
[junit4:junit4] 2> 11369 T208 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:38025_he_faz%2Fhj",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:38025/he_faz/hj"}
[junit4:junit4] 2> 11369 T208 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 11370 T208 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4:junit4] 2> 11378 T213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 11378 T256 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 11379 T207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 11378 T226 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 11378 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 11457 T257 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 11457 T257 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:38025/he_faz/hj collection:collection1 shard:shard3
[junit4:junit4] 2> 11459 T257 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4:junit4] 2> 11471 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d000a type:delete cxid:0x42 zxid:0x9d txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders/shard3 Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders/shard3
[junit4:junit4] 2> 11472 T257 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 11473 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d000a type:create cxid:0x43 zxid:0x9e txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 11477 T257 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 11477 T257 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 11478 T257 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:38025/he_faz/hj/collection1/
[junit4:junit4] 2> 11478 T257 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 11479 T257 oasc.SyncStrategy.syncToMe http://127.0.0.1:38025/he_faz/hj/collection1/ has no replicas
[junit4:junit4] 2> 11479 T257 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:38025/he_faz/hj/collection1/
[junit4:junit4] 2> 11480 T257 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4:junit4] 2> 11487 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d000a type:create cxid:0x4c zxid:0xa1 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 12892 T208 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 12907 T213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 12908 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 12908 T226 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 12908 T207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 12908 T256 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 12945 T257 oasc.ZkController.register We are http://127.0.0.1:38025/he_faz/hj/collection1/ and leader is http://127.0.0.1:38025/he_faz/hj/collection1/
[junit4:junit4] 2> 12945 T257 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:38025/he_faz/hj
[junit4:junit4] 2> 12945 T257 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 12946 T257 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 12946 T257 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 12948 T257 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 12950 T184 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 12950 T184 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 12950 T184 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 13018 T184 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 13021 T184 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:51111
[junit4:junit4] 2> 13022 T184 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 13023 T184 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 13023 T184 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1360270477354
[junit4:junit4] 2> 13024 T184 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1360270477354/solr.xml
[junit4:junit4] 2> 13024 T184 oasc.CoreContainer.<init> New CoreContainer 684481644
[junit4:junit4] 2> 13025 T184 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1360270477354/'
[junit4:junit4] 2> 13025 T184 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1360270477354/'
[junit4:junit4] 2> 13045 T184 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 60000
[junit4:junit4] 2> 13046 T184 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 13046 T184 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 13046 T184 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 13047 T184 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 13047 T184 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 13047 T184 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 13048 T184 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 13048 T184 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 13048 T184 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=60000&connTimeout=15000&retry=false
[junit4:junit4] 2> 13055 T184 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 13065 T184 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:35545/solr
[junit4:junit4] 2> 13066 T184 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 13066 T184 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35545 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@3cf94dad
[junit4:junit4] 2> 13067 T184 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 13067 T268 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:35545. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 13068 T268 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:35545, initiating session
[junit4:junit4] 2> 13068 T186 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:52505
[junit4:junit4] 2> 13069 T186 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:52505
[junit4:junit4] 2> 13070 T188 oazs.ZooKeeperServer.finishSessionInit Established session 0x13cb670159d000b with negotiated timeout 20000 for client /127.0.0.1:52505
[junit4:junit4] 2> 13070 T268 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:35545, sessionid = 0x13cb670159d000b, negotiated timeout = 20000
[junit4:junit4] 2> 13071 T269 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3cf94dad name:ZooKeeperConnection Watcher:127.0.0.1:35545 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 13071 T184 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 13073 T189 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13cb670159d000b
[junit4:junit4] 2> 13074 T186 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:52505 which had sessionid 0x13cb670159d000b
[junit4:junit4] 2> 13073 T184 oaz.ZooKeeper.close Session: 0x13cb670159d000b closed
[junit4:junit4] 2> 13074 T269 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 13074 T184 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=60000&connTimeout=15000
[junit4:junit4] 2> 13077 T184 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35545/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@6c506ee7
[junit4:junit4] 2> 13078 T184 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 13078 T270 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:35545. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 13079 T270 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:35545, initiating session
[junit4:junit4] 2> 13079 T186 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:52506
[junit4:junit4] 2> 13080 T186 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:52506
[junit4:junit4] 2> 13082 T188 oazs.ZooKeeperServer.finishSessionInit Established session 0x13cb670159d000c with negotiated timeout 20000 for client /127.0.0.1:52506
[junit4:junit4] 2> 13082 T270 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:35545, sessionid = 0x13cb670159d000c, negotiated timeout = 20000
[junit4:junit4] 2> 13083 T271 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6c506ee7 name:ZooKeeperConnection Watcher:127.0.0.1:35545/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 13084 T184 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 13085 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d000c type:create cxid:0x1 zxid:0xae txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 13086 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d000c type:create cxid:0x2 zxid:0xaf txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 13087 T184 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51111_he_faz%2Fhj
[junit4:junit4] 2> 13088 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d000c type:delete cxid:0x4 zxid:0xb0 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:51111_he_faz%2Fhj Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:51111_he_faz%2Fhj
[junit4:junit4] 2> 13088 T184 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51111_he_faz%2Fhj
[junit4:junit4] 2> 13091 T207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 13091 T226 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 13092 T213 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 13093 T241 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 13092 T256 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 13093 T213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 13093 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 13093 T256 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 13095 T226 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 13095 T207 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 13097 T184 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 13101 T272 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1360270477354/collection1
[junit4:junit4] 2> 13102 T272 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 13102 T272 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 13103 T272 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 13104 T272 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1360270477354/collection1/'
[junit4:junit4] 2> 13104 T272 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1360270477354/collection1/lib/README' to classloader
[junit4:junit4] 2> 13105 T272 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1360270477354/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 13138 T272 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 13182 T272 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 13183 T272 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 13186 T272 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 13492 T272 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 13498 T272 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 13501 T272 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 13514 T272 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 13517 T272 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 13520 T272 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 13522 T272 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 13523 T272 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1360270477354/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty4/
[junit4:junit4] 2> 13523 T272 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@f5bfdbd
[junit4:junit4] 2> 13523 T272 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 13524 T272 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty4 forceNew:false
[junit4:junit4] 2> 13524 T272 oasc.CachingDirectoryFactory.close Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty4
[junit4:junit4] 2> 13524 T272 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty4/index/
[junit4:junit4] 2> 13525 T272 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 13525 T272 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty4/index forceNew:false
[junit4:junit4] 2> 13528 T272 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=RateLimitedDirectoryWrapper(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@31664cde),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 13529 T272 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 13529 T272 oasc.CachingDirectoryFactory.close Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty4/index
[junit4:junit4] 2> 13529 T272 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4] 2> 13531 T272 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 13531 T272 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4] 2> 13531 T272 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 13532 T272 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4] 2> 13532 T272 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 13533 T272 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 13533 T272 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 13533 T272 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 13534 T272 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 13534 T272 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4] 2> 13535 T272 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4] 2> 13535 T272 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 13536 T272 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4] 2> 13536 T272 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4] 2> 13536 T272 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4] 2> 13537 T272 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 13537 T272 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 13538 T272 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 13538 T272 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 13539 T272 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 13539 T272 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 13540 T272 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 13540 T272 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 13541 T272 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 13541 T272 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4] 2> 13542 T272 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4] 2> 13543 T272 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 13545 T272 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4] 2> 13546 T272 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4] 2> 13548 T272 oasc.CachingDirectoryFactory.close Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty4
[junit4:junit4] 2> 13549 T272 oass.SolrIndexSearcher.<init> Opening Searcher@e13b792 main
[junit4:junit4] 2> 13549 T272 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 13549 T272 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 13550 T272 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4] 2> 13556 T272 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4] 2> 13581 T273 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@e13b792 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 13588 T272 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 13589 T272 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 14419 T208 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 14421 T208 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:38025_he_faz%2Fhj",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:38025/he_faz/hj"}
[junit4:junit4] 2> 14428 T208 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:51111_he_faz%2Fhj",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:51111/he_faz/hj"}
[junit4:junit4] 2> 14429 T208 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 14429 T208 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 14438 T213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 14438 T226 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 14438 T256 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 14438 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 14438 T207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 14438 T271 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 14590 T272 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 14591 T272 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51111/he_faz/hj collection:collection1 shard:shard1
[junit4:junit4] 2> 14597 T272 oasc.ZkController.register We are http://127.0.0.1:51111/he_faz/hj/collection1/ and leader is http://127.0.0.1:36781/he_faz/hj/collection1/
[junit4:junit4] 2> 14597 T272 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51111/he_faz/hj
[junit4:junit4] 2> 14598 T272 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 14598 T272 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C54 name=collection1 org.apache.solr.core.SolrCore@12699df5 url=http://127.0.0.1:51111/he_faz/hj/collection1 node=127.0.0.1:51111_he_faz%2Fhj C54_STATE=coll:collection1 core:collection1 props:{shard=null, roles=null, state=down, core=collection1, collection=collection1, node_name=127.0.0.1:51111_he_faz%2Fhj, base_url=http://127.0.0.1:51111/he_faz/hj}
[junit4:junit4] 2> 14603 T274 C54 P51111 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 14604 T274 C54 P51111 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 14604 T272 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 14604 T274 C54 P51111 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 14605 T274 C54 P51111 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 14606 T184 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 14607 T274 C54 P51111 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 14607 T184 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 14608 T184 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 14670 T184 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 14673 T184 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57532
[junit4:junit4] 2> 14673 T184 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 14674 T184 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 14674 T184 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1360270479016
[junit4:junit4] 2> 14674 T184 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1360270479016/solr.xml
[junit4:junit4] 2> 14675 T184 oasc.CoreContainer.<init> New CoreContainer 689962921
[junit4:junit4] 2> 14675 T184 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1360270479016/'
[junit4:junit4] 2> 14675 T184 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1360270479016/'
[junit4:junit4] 2> 14713 T184 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 60000
[junit4:junit4] 2> 14714 T184 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 14714 T184 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 14714 T184 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 14715 T184 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 14715 T184 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 14715 T184 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 14715 T184 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 14716 T184 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 14716 T184 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=60000&connTimeout=15000&retry=false
[junit4:junit4] 2> 14722 T184 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 14732 T184 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:35545/solr
[junit4:junit4] 2> 14732 T184 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 14733 T184 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35545 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@431d9f05
[junit4:junit4] 2> 14734 T184 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 14734 T284 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:35545. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 14735 T284 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:35545, initiating session
[junit4:junit4] 2> 14735 T186 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:52512
[junit4:junit4] 2> 14735 T186 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:52512
[junit4:junit4] 2> 14736 T188 oazs.ZooKeeperServer.finishSessionInit Established session 0x13cb670159d000d with negotiated timeout 20000 for client /127.0.0.1:52512
[junit4:junit4] 2> 14736 T284 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:35545, sessionid = 0x13cb670159d000d, negotiated timeout = 20000
[junit4:junit4] 2> 14736 T285 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@431d9f05 name:ZooKeeperConnection Watcher:127.0.0.1:35545 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 14737 T184 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 14738 T189 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13cb670159d000d
[junit4:junit4] 2> 14738 T285 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 14738 T184 oaz.ZooKeeper.close Session: 0x13cb670159d000d closed
[junit4:junit4] 2> 14738 T186 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:52512 which had sessionid 0x13cb670159d000d
[junit4:junit4] 2> 14739 T184 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=60000&connTimeout=15000
[junit4:junit4] 2> 14742 T184 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35545/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@5c382de5
[junit4:junit4] 2> 14742 T184 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 14743 T286 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:35545. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 14743 T286 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:35545, initiating session
[junit4:junit4] 2> 14744 T186 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:52514
[junit4:junit4] 2> 14744 T186 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:52514
[junit4:junit4] 2> 14745 T188 oazs.ZooKeeperServer.finishSessionInit Established session 0x13cb670159d000e with negotiated timeout 20000 for client /127.0.0.1:52514
[junit4:junit4] 2> 14745 T286 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:35545, sessionid = 0x13cb670159d000e, negotiated timeout = 20000
[junit4:junit4] 2> 14745 T287 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5c382de5 name:ZooKeeperConnection Watcher:127.0.0.1:35545/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 14745 T184 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 14746 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d000e type:create cxid:0x1 zxid:0xc0 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 14747 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d000e type:create cxid:0x2 zxid:0xc1 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 14748 T184 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57532_he_faz%2Fhj
[junit4:junit4] 2> 14748 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d000e type:delete cxid:0x4 zxid:0xc2 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:57532_he_faz%2Fhj Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:57532_he_faz%2Fhj
[junit4:junit4] 2> 14749 T184 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57532_he_faz%2Fhj
[junit4:junit4] 2> 14750 T271 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 14751 T226 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 14751 T207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 14751 T241 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 14751 T256 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 14751 T213 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 14752 T213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 14751 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 14752 T256 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 14753 T271 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 14753 T207 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 14754 T226 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 14755 T184 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 14760 T288 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1360270479016/collection1
[junit4:junit4] 2> 14760 T288 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 14761 T288 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 14761 T288 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 14762 T288 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1360270479016/collection1/'
[junit4:junit4] 2> 14763 T288 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1360270479016/collection1/lib/README' to classloader
[junit4:junit4] 2> 14763 T288 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1360270479016/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 14947 T288 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 14985 T288 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 14986 T288 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 14990 T288 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 15302 T288 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 15307 T288 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 15309 T288 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 15320 T288 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 15323 T288 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 15326 T288 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 15327 T288 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 15328 T288 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1360270479016/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty5/
[junit4:junit4] 2> 15328 T288 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@f5bfdbd
[junit4:junit4] 2> 15329 T288 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 15329 T288 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty5 forceNew:false
[junit4:junit4] 2> 15330 T288 oasc.CachingDirectoryFactory.close Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty5
[junit4:junit4] 2> 15330 T288 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty5/index/
[junit4:junit4] 2> 15330 T288 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty5/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 15330 T288 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty5/index forceNew:false
[junit4:junit4] 2> 15333 T288 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=RateLimitedDirectoryWrapper(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty5/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@415ed7e7),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 15333 T288 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 15334 T288 oasc.CachingDirectoryFactory.close Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty5/index
[junit4:junit4] 2> 15334 T288 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4] 2> 15335 T288 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 15335 T288 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4] 2> 15336 T288 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 15336 T288 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4] 2> 15336 T288 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 15337 T288 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 15337 T288 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 15337 T288 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 15338 T288 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 15338 T288 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4] 2> 15339 T288 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4] 2> 15339 T288 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 15340 T288 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4] 2> 15340 T288 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4] 2> 15340 T288 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4] 2> 15341 T288 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 15341 T288 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 15341 T288 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 15342 T288 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 15342 T288 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 15343 T288 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 15343 T288 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 15344 T288 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 15344 T288 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 15345 T288 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4] 2> 15345 T288 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4] 2> 15347 T288 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 15348 T288 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4] 2> 15349 T288 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4] 2> 15350 T288 oasc.CachingDirectoryFactory.close Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty5
[junit4:junit4] 2> 15351 T288 oass.SolrIndexSearcher.<init> Opening Searcher@3535a92b main
[junit4:junit4] 2> 15352 T288 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 15352 T288 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 15352 T288 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4] 2> 15356 T288 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4] 2> 15375 T289 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3535a92b main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 15380 T288 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 15380 T288 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 15951 T208 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 15953 T208 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:51111_he_faz%2Fhj",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:51111/he_faz/hj"}
[junit4:junit4] 2> 15961 T208 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:57532_he_faz%2Fhj",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:57532/he_faz/hj"}
[junit4:junit4] 2> 15961 T208 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 15962 T208 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4:junit4] 2> 15970 T271 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 15970 T213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 15970 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 15970 T256 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 15970 T207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 15971 T226 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 15971 T287 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 16381 T288 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 16382 T288 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:57532/he_faz/hj collection:collection1 shard:shard2
[junit4:junit4] 2> 16387 T288 oasc.ZkController.register We are http://127.0.0.1:57532/he_faz/hj/collection1/ and leader is http://127.0.0.1:49471/he_faz/hj/collection1/
[junit4:junit4] 2> 16387 T288 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:57532/he_faz/hj
[junit4:junit4] 2> 16388 T288 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 16388 T288 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C55 name=collection1 org.apache.solr.core.SolrCore@17da89a0 url=http://127.0.0.1:57532/he_faz/hj/collection1 node=127.0.0.1:57532_he_faz%2Fhj C55_STATE=coll:collection1 core:collection1 props:{shard=null, roles=null, state=down, core=collection1, collection=collection1, node_name=127.0.0.1:57532_he_faz%2Fhj, base_url=http://127.0.0.1:57532/he_faz/hj}
[junit4:junit4] 2> 16389 T290 C55 P57532 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 16390 T290 C55 P57532 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 16390 T288 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 16390 T290 C55 P57532 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 16391 T290 C55 P57532 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 16392 T184 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 16392 T290 C55 P57532 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 16393 T184 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 16394 T184 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 16459 T184 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 16462 T184 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:35726
[junit4:junit4] 2> 16462 T184 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 16463 T184 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 16463 T184 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1360270480802
[junit4:junit4] 2> 16464 T184 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1360270480802/solr.xml
[junit4:junit4] 2> 16465 T184 oasc.CoreContainer.<init> New CoreContainer 943447228
[junit4:junit4] 2> 16465 T184 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1360270480802/'
[junit4:junit4] 2> 16466 T184 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1360270480802/'
[junit4:junit4] 2> 16484 T184 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 60000
[junit4:junit4] 2> 16484 T184 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 16484 T184 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 16485 T184 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 16485 T184 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 16485 T184 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 16485 T184 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 16486 T184 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 16486 T184 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 16486 T184 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=60000&connTimeout=15000&retry=false
[junit4:junit4] 2> 16492 T184 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 16501 T184 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:35545/solr
[junit4:junit4] 2> 16502 T184 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 16502 T184 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35545 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@4ceafe8d
[junit4:junit4] 2> 16503 T184 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 16503 T300 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:35545. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 16504 T300 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:35545, initiating session
[junit4:junit4] 2> 16504 T186 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:52518
[junit4:junit4] 2> 16504 T186 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:52518
[junit4:junit4] 2> 16505 T188 oazs.ZooKeeperServer.finishSessionInit Established session 0x13cb670159d000f with negotiated timeout 20000 for client /127.0.0.1:52518
[junit4:junit4] 2> 16505 T300 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:35545, sessionid = 0x13cb670159d000f, negotiated timeout = 20000
[junit4:junit4] 2> 16506 T301 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4ceafe8d name:ZooKeeperConnection Watcher:127.0.0.1:35545 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 16506 T184 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 16507 T189 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13cb670159d000f
[junit4:junit4] 2> 16508 T301 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 16508 T186 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:52518 which had sessionid 0x13cb670159d000f
[junit4:junit4] 2> 16508 T184 oaz.ZooKeeper.close Session: 0x13cb670159d000f closed
[junit4:junit4] 2> 16509 T184 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=60000&connTimeout=15000
[junit4:junit4] 2> 16512 T184 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35545/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@95c19d7
[junit4:junit4] 2> 16512 T184 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 16513 T302 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:35545. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 16513 T302 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:35545, initiating session
[junit4:junit4] 2> 16514 T186 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:52519
[junit4:junit4] 2> 16514 T186 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:52519
[junit4:junit4] 2> 16515 T188 oazs.ZooKeeperServer.finishSessionInit Established session 0x13cb670159d0010 with negotiated timeout 20000 for client /127.0.0.1:52519
[junit4:junit4] 2> 16515 T302 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:35545, sessionid = 0x13cb670159d0010, negotiated timeout = 20000
[junit4:junit4] 2> 16516 T303 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@95c19d7 name:ZooKeeperConnection Watcher:127.0.0.1:35545/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 16516 T184 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 16517 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d0010 type:create cxid:0x1 zxid:0xd2 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 16518 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d0010 type:create cxid:0x2 zxid:0xd3 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 16519 T184 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35726_he_faz%2Fhj
[junit4:junit4] 2> 16520 T189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13cb670159d0010 type:delete cxid:0x4 zxid:0xd4 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:35726_he_faz%2Fhj Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:35726_he_faz%2Fhj
[junit4:junit4] 2> 16521 T184 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35726_he_faz%2Fhj
[junit4:junit4] 2> 16524 T271 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 16524 T207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 16524 T226 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 16525 T241 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 16525 T213 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 16526 T256 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 16526 T287 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 16526 T241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 16526 T256 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 16526 T213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 16526 T287 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 16528 T207 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 16529 T226 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 16529 T271 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 16532 T184 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 16541 T304 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1360270480802/collection1
[junit4:junit4] 2> 16541 T304 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 16542 T304 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 16543 T304 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 16545 T304 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1360270480802/collection1/'
[junit4:junit4] 2> 16546 T304 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1360270480802/collection1/lib/README' to classloader
[junit4:junit4] 2> 16547 T304 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1360270480802/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 16600 T304 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 16639 T304 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 16640 T304 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 16644 T304 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 17166 T304 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 17171 T304 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 17173 T304 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 17194 T304 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 17201 T304 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 17207 T304 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 17210 T304 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 17213 T304 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1360270480802/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty6/
[junit4:junit4] 2> 17213 T304 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@f5bfdbd
[junit4:junit4] 2> 17214 T304 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 17215 T304 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty6 forceNew:false
[junit4:junit4] 2> 17216 T304 oasc.CachingDirectoryFactory.close Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty6
[junit4:junit4] 2> 17216 T304 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty6/index/
[junit4:junit4] 2> 17217 T304 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty6/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 17217 T304 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty6/index forceNew:false
[junit4:junit4] 2> 17221 T304 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=RateLimitedDirectoryWrapper(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty6/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@79930089),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 17221 T304 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 17221 T304 oasc.CachingDirectoryFactory.close Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty6/index
[junit4:junit4] 2> 17221 T304 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4] 2> 17223 T304 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 17224 T304 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4] 2> 17224 T304 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 17224 T304 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4] 2> 17225 T304 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 17225 T304 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 17225 T304 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 17226 T304 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 17226 T304 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 17227 T304 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4] 2> 17227 T304 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4] 2> 17227 T304 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 17228 T304 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4] 2> 17228 T304 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4] 2> 17228 T304 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4] 2> 17229 T304 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 17229 T304 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 17230 T304 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 17231 T304 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 17231 T304 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 17232 T304 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 17232 T304 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 17233 T304 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 17233 T304 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4] 2> 17234 T304 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4] 2> 17235 T304 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4] 2> 17237 T304 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 17238 T304 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4] 2> 17240 T304 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4] 2> 17241 T304 oasc.CachingDirectoryFactory.close Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1360270464410/jetty6
[junit4:junit4] 2> 17242 T304 oass.SolrIndexSearcher.<init> Opening Searcher@6281635b main
[junit4:junit4] 2> 17242 T304 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 17243 T304 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 17243 T304 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4] 2> 17247 T304 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4] 2> 17274 T305 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6281635b main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 17284 T304 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 17284 T304 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 17481 T208 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 17482 T208 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:57532_he_faz%2Fhj",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:57532/he_faz/hj"}
[junit4:junit4] 2> 17488 T208 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "op
[...truncated too long message...]
.0.1:51111, state=BLOCKED, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4:junit4] 2> at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:102)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:662)
[junit4:junit4] 2> 58) Thread[id=1372, name=Thread-137-SendThread(localhost.localdomain:35545), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4:junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
[junit4:junit4] 2> 59) Thread[id=999, name=Thread-137-SendThread(localhost.localdomain:35545), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4:junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4:junit4] 2> at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
[junit4:junit4] 2> 60) Thread[id=1385, name=Thread-137-SendThread(localhost.localdomain:35545), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4:junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4:junit4] 2> at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
[junit4:junit4] 2> 61) Thread[id=1347, name=qtp1193997100-1347 Selector0, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4:junit4] 2> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
[junit4:junit4] 2> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
[junit4:junit4] 2> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
[junit4:junit4] 2> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
[junit4:junit4] 2> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:662)
[junit4:junit4] 2> 62) Thread[id=240, name=TEST-ChaosMonkeySafeLeaderTest.testDistribSearch-seed#[20887FE6FE7A246D]-SendThread(localhost.localdomain:35545), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4:junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4:junit4] 2> at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
[junit4:junit4] 2> 63) Thread[id=509, name=qtp1207205377-509 Acceptor0 SelectChannelConnector@127.0.0.1:42596, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4:junit4] 2> at org.eclipse.jetty.server.nio.SelectChannelConnector.getConnection(SelectChannelConnector.java:160)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:934)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:662)
[junit4:junit4] 2> 64) Thread[id=1175, name=qtp1072990022-1175 Acceptor1 SelectChannelConnector@127.0.0.1:40186, state=BLOCKED, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:956)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:662)
[junit4:junit4] 2> 7225097 T793 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:35545. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 7225097 T1325 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:35545. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 7225098 T1325 oaz.ClientCnxn$SendThread.run WARNING Session 0x13cb670159d0047 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
[junit4:junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4:junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
[junit4:junit4] 2>
[junit4:junit4] 2> 7225098 T793 oaz.ClientCnxn$SendThread.run WARNING Session 0x13cb670159d003b for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
[junit4:junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4:junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
[junit4:junit4] 2>
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {range_facet_l=PostingsFormat(name=TestBloomFilteredLucene41Postings), text=PostingsFormat(name=MockRandom), _version_=MockFixedIntBlock(blockSize=1370), rnd_b=MockVariableIntBlock(baseBlockSize=90), intDefault=MockFixedIntBlock(blockSize=1370), id=MockVariableIntBlock(baseBlockSize=90), timestamp=PostingsFormat(name=TestBloomFilteredLucene41Postings), a_t=PostingsFormat(name=TestBloomFilteredLucene41Postings), range_facet_sl=MockVariableIntBlock(baseBlockSize=90), range_facet_si=PostingsFormat(name=TestBloomFilteredLucene41Postings), other_tl1=MockFixedIntBlock(blockSize=1370), multiDefault=PostingsFormat(name=TestBloomFilteredLucene41Postings), a_si=MockFixedIntBlock(blockSize=1370)}, sim=DefaultSimilarity, locale=es_US, timezone=Indian/Kerguelen
[junit4:junit4] 2> NOTE: Linux 3.2.0-37-generic amd64/Sun Microsystems Inc. 1.6.0_38 (64-bit)/cpus=8,threads=79,free=189688704,total=312147968
[junit4:junit4] 2> NOTE: All tests run in this JVM: [SpatialFilterTest, TestAtomicUpdateErrorCases, BadComponentTest, ClusterStateUpdateTest, TestIBSimilarityFactory, ZkControllerTest, BinaryUpdateRequestHandlerTest, TestSearchPerf, FastVectorHighlighterTest, TermsComponentTest, TestComponentsName, TestCoreContainer, TestFastLRUCache, TestReload, IndexBasedSpellCheckerTest, SimplePostToolTest, FileBasedSpellCheckerTest, ChaosMonkeySafeLeaderTest]
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=20887FE6FE7A246D -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_US -Dtests.timezone=Indian/Kerguelen -Dtests.file.encoding=UTF-8
[junit4:junit4] ERROR 0.00s J0 | ChaosMonkeySafeLeaderTest (suite) <<<
[junit4:junit4] > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([20887FE6FE7A246D]:0)
[junit4:junit4] Completed on J0 in 7225.13s, 1 test, 2 errors <<< FAILURES!
[...truncated 140 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:361: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:183: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1176: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:841: There were test failures: 249 suites, 1038 tests, 1 suite-level error, 1 error, 181 ignored (1 assumption)
Total time: 137 minutes 2 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Description set: Java: 64bit/jdk1.6.0_38 -XX:+UseParallelGC
Email was triggered for: Failure
Sending email for trigger: Failure