You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2014/11/24 14:18:04 UTC

[JENKINS] Lucene-Solr-Tests-5.x-Java7 - Build # 2228 - Still Failing

Build: https://builds.apache.org/job/Lucene-Solr-Tests-5.x-Java7/2228/

All tests passed

Build Log:
[...truncated 61125 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/build.xml:525: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/build.xml:432: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/extra-targets.xml:105: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/extra-targets.xml:204: The following files are missing svn:eol-style (or binary svn:mime-type):
* ./solr/server/solr/configsets/data_driven_schema_configs/conf/velocity/VM_global_library.vm
* ./solr/server/solr/configsets/data_driven_schema_configs/conf/velocity/browse.vm
* ./solr/server/solr/configsets/data_driven_schema_configs/conf/velocity/debug.vm
* ./solr/server/solr/configsets/data_driven_schema_configs/conf/velocity/error.vm
* ./solr/server/solr/configsets/data_driven_schema_configs/conf/velocity/facets.vm
* ./solr/server/solr/configsets/data_driven_schema_configs/conf/velocity/footer.vm
* ./solr/server/solr/configsets/data_driven_schema_configs/conf/velocity/head.vm
* ./solr/server/solr/configsets/data_driven_schema_configs/conf/velocity/header.vm
* ./solr/server/solr/configsets/data_driven_schema_configs/conf/velocity/hit.vm
* ./solr/server/solr/configsets/data_driven_schema_configs/conf/velocity/jquery.autocomplete.css
* ./solr/server/solr/configsets/data_driven_schema_configs/conf/velocity/jquery.autocomplete.js
* ./solr/server/solr/configsets/data_driven_schema_configs/conf/velocity/layout.vm
* ./solr/server/solr/configsets/data_driven_schema_configs/conf/velocity/main.css
* ./solr/server/solr/configsets/data_driven_schema_configs/conf/velocity/pagination_bottom.vm
* ./solr/server/solr/configsets/data_driven_schema_configs/conf/velocity/pagination_top.vm
* ./solr/server/solr/configsets/data_driven_schema_configs/conf/velocity/results_list.vm

Total time: 110 minutes 54 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-Tests-5.x-Java7 #2226
Archived 1 artifacts
Archive block size is 32768
Received 0 blocks and 464 bytes
Compression is 0.0%
Took 25 ms
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-5.x-Java7 - Build # 2233 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-5.x-Java7/2233/

2 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([BC77A3F2BD455FD1]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([BC77A3F2BD455FD1]:0)




Build Log:
[...truncated 12391 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/init-core-data-001
   [junit4]   2> 2193718 T5436 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 2193719 T5436 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /vk_c/w
   [junit4]   2> 2193724 T5436 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 2193724 T5436 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2193725 T5437 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2193825 T5436 oasc.ZkTestServer.run start zk server on port:60825
   [junit4]   2> 2193826 T5436 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2193827 T5436 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2193830 T5444 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3009f282 name:ZooKeeperConnection Watcher:127.0.0.1:60825 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2193831 T5436 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2193831 T5436 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2193831 T5436 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2193833 T5436 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2193834 T5436 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2193835 T5447 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5c0fbe7c name:ZooKeeperConnection Watcher:127.0.0.1:60825/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2193836 T5436 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2193836 T5436 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2193836 T5436 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 2193838 T5436 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 2193852 T5436 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 2193854 T5436 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 2193855 T5436 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2193856 T5436 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 2193858 T5436 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 2193859 T5436 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 2193961 T5436 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2193962 T5436 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2193963 T5436 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2193964 T5436 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 2193966 T5436 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2193966 T5436 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 2193968 T5436 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2193969 T5436 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 2193970 T5436 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2193971 T5436 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 2193973 T5436 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2193973 T5436 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2193975 T5436 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2193976 T5436 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2193977 T5436 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2193978 T5436 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 2193979 T5436 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2193980 T5436 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 2195489 T5436 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2195493 T5436 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:60831
   [junit4]   2> 2195494 T5436 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2195494 T5436 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2195494 T5436 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-002
   [junit4]   2> 2195494 T5436 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-002/'
   [junit4]   2> 2195537 T5436 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-002/solr.xml
   [junit4]   2> 2195612 T5436 oasc.CoreContainer.<init> New CoreContainer 854772087
   [junit4]   2> 2195613 T5436 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-002/]
   [junit4]   2> 2195614 T5436 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2195614 T5436 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2195614 T5436 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2195614 T5436 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2195615 T5436 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 2195615 T5436 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2195615 T5436 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2195615 T5436 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2195616 T5436 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2195616 T5436 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2195616 T5436 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2195617 T5436 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2195617 T5436 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2195617 T5436 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2195617 T5436 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60825/solr
   [junit4]   2> 2195618 T5436 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2195618 T5436 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2195619 T5436 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2195620 T5459 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@128a363f name:ZooKeeperConnection Watcher:127.0.0.1:60825 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2195621 T5436 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2195621 T5436 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2195622 T5436 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2195623 T5462 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ac00985 name:ZooKeeperConnection Watcher:127.0.0.1:60825/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2195624 T5436 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2195624 T5436 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2195626 T5436 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2195627 T5436 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 2195629 T5436 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 2195630 T5436 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 2195632 T5436 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2195632 T5436 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60831_vk_c%2Fw
   [junit4]   2> 2195633 T5436 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60831_vk_c%2Fw
   [junit4]   2> 2195635 T5436 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2195636 T5436 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2195637 T5436 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 2195638 T5436 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:60831_vk_c%2Fw
   [junit4]   2> 2195638 T5436 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2195639 T5436 oasc.Overseer.start Overseer (id=92859329149927427-127.0.0.1:60831_vk_c%2Fw-n_0000000000) starting
   [junit4]   2> 2195640 T5436 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2195645 T5436 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 2195645 T5464 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2195645 T5436 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2195647 T5436 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2195648 T5436 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2195650 T5463 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2195653 T5466 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 2195654 T5466 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2195654 T5466 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2195654 T5462 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2195656 T5463 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:60831/vk_c/w",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "node_name":"127.0.0.1:60831_vk_c%2Fw",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "state":"down"}
   [junit4]   2> 2195656 T5463 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 2195656 T5463 oasc.Overseer$ClusterStateUpdater.createCollection state version control_collection 1
   [junit4]   2> 2195656 T5463 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2195658 T5462 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2196655 T5466 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2196656 T5466 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 2196656 T5466 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2196657 T5466 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 2196657 T5466 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2196658 T5466 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-002/collection1/'
   [junit4]   2> 2196659 T5466 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 2196660 T5466 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 2196660 T5466 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 2196693 T5466 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 2196695 T5466 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 2196729 T5466 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 2196776 T5466 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2196877 T5466 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 2196888 T5466 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2197246 T5466 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2197249 T5466 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2197251 T5466 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2197256 T5466 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2197273 T5466 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 2197273 T5466 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 2197274 T5466 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-002/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-001/control/data/
   [junit4]   2> 2197274 T5466 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b9af1f7
   [junit4]   2> 2197277 T5466 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-001/control/data
   [junit4]   2> 2197278 T5466 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-001/control/data/index/
   [junit4]   2> 2197279 T5466 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2197280 T5466 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-001/control/data/index
   [junit4]   2> 2197280 T5466 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=42, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5601765203538792]
   [junit4]   2> 2197286 T5466 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 2197287 T5466 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2197292 T5466 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2197292 T5466 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2197292 T5466 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2197293 T5466 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2197293 T5466 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2197293 T5466 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2197294 T5466 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2197294 T5466 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2197294 T5466 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2197295 T5466 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2197295 T5466 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 2197296 T5466 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2197296 T5466 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2197297 T5466 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2197297 T5466 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2197298 T5466 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2197298 T5466 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2197298 T5466 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2197299 T5466 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2197299 T5466 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2197300 T5466 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2197319 T5466 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2197322 T5466 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2197324 T5466 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2197325 T5466 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2197329 T5466 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2197329 T5466 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2197330 T5466 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=10, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.294543358993952]
   [junit4]   2> 2197332 T5466 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 2197332 T5466 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2197333 T5466 oass.SolrIndexSearcher.<init> Opening Searcher@25a822c1[collection1] main
   [junit4]   2> 2197333 T5466 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 2197334 T5466 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2197334 T5466 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2197334 T5466 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2197335 T5466 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2197335 T5466 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2197335 T5466 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2197336 T5466 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2197336 T5466 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 2197338 T5466 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2197338 T5466 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2197338 T5466 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2197341 T5466 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2197342 T5467 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@25a822c1[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2197342 T5470 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:60831/vk_c/w collection:control_collection shard:shard1
   [junit4]   2> 2197342 T5436 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1
   [junit4]   2> 2197343 T5436 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2197343 T5470 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 2197344 T5436 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2197345 T5436 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2197346 T5473 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@34bc0283 name:ZooKeeperConnection Watcher:127.0.0.1:60825/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2197346 T5436 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2197347 T5436 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2197348 T5436 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2197348 T5470 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 2197349 T5462 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2197349 T5470 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2197350 T5470 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C364 name=collection1 org.apache.solr.core.SolrCore@5576550f url=http://127.0.0.1:60831/vk_c/w/collection1 node=127.0.0.1:60831_vk_c%2Fw C364_STATE=coll:control_collection core:collection1 props:{base_url=http://127.0.0.1:60831/vk_c/w, core=collection1, node_name=127.0.0.1:60831_vk_c%2Fw, state=down}
   [junit4]   2> 2197350 T5470 C364 P60831 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:60831/vk_c/w/collection1/
   [junit4]   2> 2197350 T5436 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2197350 T5470 C364 P60831 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2197351 T5436 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 2197351 T5470 C364 P60831 oasc.SyncStrategy.syncToMe http://127.0.0.1:60831/vk_c/w/collection1/ has no replicas
   [junit4]   2> 2197351 T5436 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2197351 T5463 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection"}
   [junit4]   2> 2197351 T5470 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:60831/vk_c/w/collection1/ shard1
   [junit4]   2> 2197352 T5436 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2197352 T5470 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 2197354 T5476 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13701f3 name:ZooKeeperConnection Watcher:127.0.0.1:60825/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2197354 T5436 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2197354 T5436 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2197355 T5473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2197355 T5462 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2197359 T5463 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 2, message = {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "name":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "stateFormat":2}
   [junit4]   2> 2197360 T5463 oasc.Overseer$ClusterStateUpdater.buildCollection Building a new collection: collection1
   [junit4]   2> 2197360 T5463 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 2197360 T5463 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2
   [junit4]   2> 2197362 T5462 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2197362 T5463 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/collection1/state.json
   [junit4]   2> 2197363 T5462 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2197363 T5473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2197366 T5463 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:60831/vk_c/w",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"}
   [junit4]   2> 2197368 T5462 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2197369 T5462 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2197369 T5473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2197408 T5470 oasc.ZkController.register We are http://127.0.0.1:60831/vk_c/w/collection1/ and leader is http://127.0.0.1:60831/vk_c/w/collection1/
   [junit4]   2> 2197409 T5470 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60831/vk_c/w
   [junit4]   2> 2197409 T5470 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2197409 T5470 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 2197409 T5470 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2197411 T5462 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2197413 T5463 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:60831/vk_c/w",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "node_name":"127.0.0.1:60831_vk_c%2Fw",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "state":"active"}
   [junit4]   2> 2197516 T5462 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2197516 T5473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2199006 T5436 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 2199007 T5436 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2199010 T5436 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:60841
   [junit4]   2> 2199010 T5436 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2199011 T5436 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2199011 T5436 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-003
   [junit4]   2> 2199011 T5436 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-003/'
   [junit4]   2> 2199061 T5436 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-003/solr.xml
   [junit4]   2> 2199161 T5436 oasc.CoreContainer.<init> New CoreContainer 1194775815
   [junit4]   2> 2199161 T5436 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-003/]
   [junit4]   2> 2199163 T5436 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2199163 T5436 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2199163 T5436 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2199163 T5436 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2199164 T5436 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 2199164 T5436 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2199164 T5436 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2199164 T5436 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2199165 T5436 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2199165 T5436 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2199165 T5436 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2199166 T5436 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2199166 T5436 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2199166 T5436 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2199166 T5436 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60825/solr
   [junit4]   2> 2199167 T5436 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2199167 T5436 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2199168 T5436 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2199169 T5488 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17f43407 name:ZooKeeperConnection Watcher:127.0.0.1:60825 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2199170 T5436 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2199170 T5436 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2199172 T5436 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2199173 T5491 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e1c1503 name:ZooKeeperConnection Watcher:127.0.0.1:60825/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2199173 T5436 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2199176 T5436 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2200180 T5436 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60841_vk_c%2Fw
   [junit4]   2> 2200181 T5436 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60841_vk_c%2Fw
   [junit4]   2> 2200184 T5436 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 2200192 T5492 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 2200192 T5492 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2200194 T5462 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2200194 T5492 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 2200194 T5492 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 2200195 T5492 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 2200196 T5492 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 0 
   [junit4]   2> 2200196 T5492 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2200198 T5463 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:60841/vk_c/w",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "node_name":"127.0.0.1:60841_vk_c%2Fw",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "state":"down"}
   [junit4]   2> 2200200 T5463 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 2200200 T5463 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 2200203 T5462 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2200203 T5463 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 2200204 T5491 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 2200205 T5491 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 1 
   [junit4]   2> 2201196 T5492 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2201197 T5492 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2201197 T5492 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2201198 T5492 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2201198 T5492 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2201199 T5492 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-003/collection1/'
   [junit4]   2> 2201201 T5492 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 2201202 T5492 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 2201204 T5492 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 2201245 T5492 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 2201248 T5492 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 2201288 T5492 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 2201343 T5492 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2201444 T5492 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 2201455 T5492 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2201725 T5492 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2201727 T5492 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2201729 T5492 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2201732 T5492 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2201746 T5492 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 2201747 T5492 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 2201747 T5492 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-003/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-001/jetty1/
   [junit4]   2> 2201747 T5492 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b9af1f7
   [junit4]   2> 2201748 T5492 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-001/jetty1
   [junit4]   2> 2201749 T5492 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-001/jetty1/index/
   [junit4]   2> 2201750 T5492 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 2201751 T5492 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-001/jetty1/index
   [junit4]   2> 2201752 T5492 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=42, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5601765203538792]
   [junit4]   2> 2201774 T5492 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 2201774 T5492 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2201781 T5492 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2201781 T5492 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2201782 T5492 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2201782 T5492 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2201782 T5492 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2201782 T5492 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2201783 T5492 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2201783 T5492 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2201784 T5492 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2201784 T5492 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2201785 T5492 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 2201786 T5492 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2201786 T5492 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2201787 T5492 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2201787 T5492 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2201788 T5492 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2201788 T5492 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2201789 T5492 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2201789 T5492 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2201790 T5492 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2201791 T5492 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2201815 T5492 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2201817 T5492 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2201819 T5492 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2201821 T5492 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2201825 T5492 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2201825 T5492 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2201827 T5492 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=10, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.294543358993952]
   [junit4]   2> 2201968 T5492 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 2201968 T5492 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2201968 T5492 oass.SolrIndexSearcher.<init> Opening Searcher@2b9481ae[collection1] main
   [junit4]   2> 2201969 T5492 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2201969 T5492 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2201969 T5492 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2201970 T5492 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2201970 T5492 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2201970 T5492 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2201971 T5492 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2201971 T5492 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2201971 T5492 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 2201971 T5492 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2201974 T5493 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2b9481ae[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2201975 T5492 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2201975 T5496 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:60841/vk_c/w collection:collection1 shard:shard3
   [junit4]   2> 2201975 T5436 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1
   [junit4]   2> 2201976 T5436 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2201976 T5496 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 2201979 T5496 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 2201980 T5462 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2201980 T5496 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2201981 T5496 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C365 name=collection1 org.apache.solr.core.SolrCore@269d3e8a url=http://127.0.0.1:60841/vk_c/w/collection1 node=127.0.0.1:60841_vk_c%2Fw C365_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:60841/vk_c/w, core=collection1, node_name=127.0.0.1:60841_vk_c%2Fw, state=down}
   [junit4]   2> 2201981 T5496 C365 P60841 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:60841/vk_c/w/collection1/
   [junit4]   2> 2201981 T5496 C365 P60841 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2201981 T5496 C365 P60841 oasc.SyncStrategy.syncToMe http://127.0.0.1:60841/vk_c/w/collection1/ has no replicas
   [junit4]   2> 2201981 T5496 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:60841/vk_c/w/collection1/ shard3
   [junit4]   2> 2201982 T5496 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 2201982 T5463 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "collection":"collection1"}
   [junit4]   2> 2201984 T5463 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 2201985 T5491 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 2201985 T5491 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 2 
   [junit4]   2> 2201988 T5463 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:60841/vk_c/w",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"}
   [junit4]   2> 2201989 T5462 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2201989 T5463 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 2201990 T5491 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 2201990 T5491 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 
   [junit4]   2> 2202034 T5496 oasc.ZkController.register We are http://127.0.0.1:60841/vk_c/w/collection1/ and leader is http://127.0.0.1:60841/vk_c/w/collection1/
   [junit4]   2> 2202035 T5496 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60841/vk_c/w
   [junit4]   2> 2202035 T5496 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2202035 T5496 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 2202035 T5496 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2202037 T5462 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2202039 T5496 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 
   [junit4]   2> 2202039 T5463 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:60841/vk_c/w",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "node_name":"127.0.0.1:60841_vk_c%2Fw",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "state":"active"}
   [junit4]   2> 2202043 T5463 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 2202044 T5491 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 2202044 T5491 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 2203149 T5436 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 2203150 T5436 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2203153 T5436 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:39961
   [junit4]   2> 2203153 T5436 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2203153 T5436 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2203154 T5436 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-004
   [junit4]   2> 2203154 T5436 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-004/'
   [junit4]   2> 2203186 T5436 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-004/solr.xml
   [junit4]   2> 2203244 T5436 oasc.CoreContainer.<init> New CoreContainer 888964795
   [junit4]   2> 2203245 T5436 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-004/]
   [junit4]   2> 2203245 T5436 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2203246 T5436 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2203246 T5436 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2203246 T5436 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2203246 T5436 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 2203247 T5436 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2203247 T5436 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2203247 T5436 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2203247 T5436 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2203248 T5436 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2203248 T5436 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2203248 T5436 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2203249 T5436 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2203249 T5436 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2203249 T5436 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60825/solr
   [junit4]   2> 2203249 T5436 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2203250 T5436 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2203250 T5436 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2203252 T5508 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@59530d2d name:ZooKeeperConnection Watcher:127.0.0.1:60825 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2203253 T5436 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2203253 T5436 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2203254 T5436 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2203256 T5511 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@49db714b name:ZooKeeperConnection Watcher:127.0.0.1:60825/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2203256 T5436 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2203259 T5436 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2204262 T5436 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39961_vk_c%2Fw
   [junit4]   2> 2204263 T5436 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39961_vk_c%2Fw
   [junit4]   2> 2204266 T5436 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 2204274 T5512 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 2204274 T5512 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2204276 T5462 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2204276 T5512 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 2204276 T5512 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 2204277 T5512 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 2204278 T5512 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 2204278 T5512 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2204278 T5463 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39961/vk_c/w",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "node_name":"127.0.0.1:39961_vk_c%2Fw",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "state":"down"}
   [junit4]   2> 2204280 T5463 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 2204281 T5463 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 2204284 T5463 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 2204285 T5511 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 2204285 T5491 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 2204286 T5511 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> 2204286 T5491 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> 2205278 T5512 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2205279 T5512 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2205279 T5512 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2205280 T5512 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2205280 T5512 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2205281 T5512 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-004/collection1/'
   [junit4]   2> 2205283 T5512 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 2205284 T5512 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 2205284 T5512 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 2205327 T5512 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 2205330 T5512 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 2205370 T5512 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 2205425 T5512 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2205526 T5512 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 2205537 T5512 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2205828 T5512 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2205830 T5512 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2205831 T5512 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2205835 T5512 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2205847 T5512 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 2205847 T5512 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 2205847 T5512 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-004/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-001/jetty2/
   [junit4]   2> 2205847 T5512 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b9af1f7
   [junit4]   2> 2205849 T5512 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-001/jetty2
   [junit4]   2> 2205849 T5512 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-001/jetty2/index/
   [junit4]   2> 2205850 T5512 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 2205851 T5512 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-001/jetty2/index
   [junit4]   2> 2205852 T5512 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=42, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5601765203538792]
   [junit4]   2> 2205882 T5512 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 2205882 T5512 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2205887 T5512 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2205887 T5512 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2205887 T5512 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2205887 T5512 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2205887 T5512 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2205888 T5512 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2205888 T5512 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2205888 T5512 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2205888 T5512 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2205889 T5512 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2205889 T5512 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 2205890 T5512 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2205890 T5512 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2205891 T5512 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2205891 T5512 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2205891 T5512 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2205891 T5512 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2205892 T5512 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2205892 T5512 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2205893 T5512 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2205893 T5512 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2205907 T5512 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2205908 T5512 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2205910 T5512 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2205911 T5512 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2205913 T5512 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2205914 T5512 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2205915 T5512 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=10, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.294543358993952]
   [junit4]   2> 2205916 T5512 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 2205916 T5512 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2205917 T5512 oass.SolrIndexSearcher.<init> Opening Searcher@6b68db18[collection1] main
   [junit4]   2> 2205917 T5512 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2205918 T5512 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2205918 T5512 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2205918 T5512 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2205918 T5512 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2205918 T5512 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2205919 T5512 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2205919 T5512 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2205919 T5512 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 2205919 T5512 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2205922 T5513 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6b68db18[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2205922 T5512 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2205923 T5516 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:39961/vk_c/w collection:collection1 shard:shard2
   [junit4]   2> 2205924 T5436 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1
   [junit4]   2> 2205924 T5436 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2205924 T5516 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 2205928 T5516 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 2205929 T5462 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2205929 T5516 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2205929 T5516 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C366 name=collection1 org.apache.solr.core.SolrCore@3f27e093 url=http://127.0.0.1:39961/vk_c/w/collection1 node=127.0.0.1:39961_vk_c%2Fw C366_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:39961/vk_c/w, core=collection1, node_name=127.0.0.1:39961_vk_c%2Fw, state=down}
   [junit4]   2> 2205929 T5516 C366 P39961 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:39961/vk_c/w/collection1/
   [junit4]   2> 2205930 T5516 C366 P39961 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2205930 T5516 C366 P39961 oasc.SyncStrategy.syncToMe http://127.0.0.1:39961/vk_c/w/collection1/ has no replicas
   [junit4]   2> 2205930 T5516 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:39961/vk_c/w/collection1/ shard2
   [junit4]   2> 2205930 T5516 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 2205931 T5463 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1"}
   [junit4]   2> 2205933 T5463 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 2205934 T5491 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 2205934 T5511 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 2205934 T5491 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 6 
   [junit4]   2> 2205934 T5511 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 6 
   [junit4]   2> 2205936 T5463 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39961/vk_c/w",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"}
   [junit4]   2> 2205938 T5462 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2205938 T5463 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 2205939 T5511 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 2205939 T5491 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 2205939 T5511 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 7 
   [junit4]   2> 2205939 T5491 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 7 
   [junit4]   2> 2205983 T5516 oasc.ZkController.register We are http://127.0.0.1:39961/vk_c/w/collection1/ and leader is http://127.0.0.1:39961/vk_c/w/collection1/
   [junit4]   2> 2205984 T5516 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:39961/vk_c/w
   [junit4]   2> 2205984 T5516 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2205984 T5516 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 2205984 T5516 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2205985 T5462 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2205987 T5516 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 7 
   [junit4]   2> 2205987 T5463 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39961/vk_c/w",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "node_name":"127.0.0.1:39961_vk_c%2Fw",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node2",
   [junit4]   2> 	  "state":"active"}
   [junit4]   2> 2205990 T5463 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 2205991 T5511 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 2205991 T5491 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 2205991 T5511 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 8 
   [junit4]   2> 2205992 T5491 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 8 
   [junit4]   2> 2207024 T5436 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 2207025 T5436 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2207027 T5436 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:22540
   [junit4]   2> 2207028 T5436 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2207028 T5436 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2207028 T5436 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-005
   [junit4]   2> 2207028 T5436 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-005/'
   [junit4]   2> 2207057 T5436 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-005/solr.xml
   [junit4]   2> 2207110 T5436 oasc.CoreContainer.<init> New CoreContainer 1749221852
   [junit4]   2> 2207110 T5436 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-BC77A3F2BD455FD1-001/tempDir-005/]
   [junit4]   2> 2207111 T5436 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2207111 T5436 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2207112 T5436 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2207112 T5436 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2207112 T5436 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 2207112 T5436 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2207113 T5436 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2207113 T5436 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2207113 T5436 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2207113 T5436 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2207114 T5436 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=f

[...truncated too long message...]

n(ClientCnxn.java:1081)
   [junit4]   2> 	  15) Thread[id=5564, name=commitScheduler-1497-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2> 	        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]   2> 	        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  16) Thread[id=5538, name=qtp1017022952-5538 Acceptor0 SocketConnector@127.0.0.1:62031, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at java.net.PlainSocketImpl.socketAccept(Native Method)
   [junit4]   2> 	        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
   [junit4]   2> 	        at java.net.ServerSocket.implAccept(ServerSocket.java:530)
   [junit4]   2> 	        at java.net.ServerSocket.accept(ServerSocket.java:498)
   [junit4]   2> 	        at org.eclipse.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
   [junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  17) Thread[id=5556, name=coreZkRegister-1498-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  18) Thread[id=6285, name=zkCallback-580-thread-115, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
   [junit4]   2> 	        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]   2> 	        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
   [junit4]   2> 	        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  19) Thread[id=5537, name=HashSessionScavenger-98, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.util.TimerThread.mainLoop(Timer.java:552)
   [junit4]   2> 	        at java.util.TimerThread.run(Timer.java:505)
   [junit4]   2> 	  20) Thread[id=5517, name=HashSessionScavenger-97, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.util.TimerThread.mainLoop(Timer.java:552)
   [junit4]   2> 	        at java.util.TimerThread.run(Timer.java:505)
   [junit4]   2> 	  21) Thread[id=5497, name=HashSessionScavenger-96, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.util.TimerThread.mainLoop(Timer.java:552)
   [junit4]   2> 	        at java.util.TimerThread.run(Timer.java:505)
   [junit4]   2> 	  22) Thread[id=5509, name=TEST-ChaosMonkeySafeLeaderTest.testDistribSearch-seed#[BC77A3F2BD455FD1]-SendThread(127.0.0.1:60825), state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
   [junit4]   2> 	        at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
   [junit4]   2> 	        at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50), sim=DefaultSimilarity, locale=sk, timezone=Canada/Atlantic
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=24,free=81474392,total=195035136
   [junit4]   2> NOTE: All tests run in this JVM: [SolrIndexConfigTest, DistribDocExpirationUpdateProcessorTest, RequestHandlersTest, TestPHPSerializedResponseWriter, ShardRoutingTest, TestSolrXmlPersistence, TestFunctionQuery, ShardRoutingCustomTest, HttpPartitionTest, TestOmitPositions, TestClassNameShortening, SuggesterFSTTest, ChangedSchemaMergeTest, HighlighterMaxOffsetTest, DocValuesMultiTest, TestNonDefinedSimilarityFactory, AnalyticsMergeStrategyTest, TestDocBasedVersionConstraints, TestSerializedLuceneMatchVersion, CoreAdminCreateDiscoverTest, TestValueSourceCache, ZkSolrClientTest, TestPartialUpdateDeduplication, TestFieldTypeResource, TestAnalyzedSuggestions, TestFuzzyAnalyzedSuggestions, TestExpandComponent, TestDistribDocBasedVersion, AliasIntegrationTest, CircularListTest, TestRestManager, TestCharFilters, PrimitiveFieldTypeTest, QueryEqualityTest, TermsComponentTest, DistributedIntervalFacetingTest, DefaultValueUpdateProcessorTest, TestFastOutputStream, DistributedFacetPivotLongTailTest, TestDistributedSearch, BadIndexSchemaTest, TestDistributedMissingSort, PreAnalyzedUpdateProcessorTest, TestQueryUtils, TermVectorComponentTest, PeerSyncTest, TestHighFrequencyDictionaryFactory, SortByFunctionTest, ScriptEngineTest, TestTolerantSearch, OverriddenZkACLAndCredentialsProvidersTest, DateMathParserTest, DirectSolrConnectionTest, ZkCLITest, LeaderFailoverAfterPartitionTest, DeleteInactiveReplicaTest, ChaosMonkeyNothingIsSafeTest, OverseerTest, TestRequestStatusCollectionAPI, ChaosMonkeySafeLeaderTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=BC77A3F2BD455FD1 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=sk -Dtests.timezone=Canada/Atlantic -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | ChaosMonkeySafeLeaderTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([BC77A3F2BD455FD1]:0)
   [junit4] Completed on J1 in 7223.96s, 1 test, 2 errors <<< FAILURES!

[...truncated 13 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/build.xml:525: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/build.xml:473: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/common-build.xml:508: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/lucene/common-build.xml:1363: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/lucene/common-build.xml:970: There were test failures: 449 suites, 1825 tests, 1 suite-level error, 1 error, 59 ignored (22 assumptions)

Total time: 183 minutes 2 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-Tests-5.x-Java7 #2226
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 189345158 bytes
Compression is 0.0%
Took 36 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-5.x-Java7 - Build # 2232 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-5.x-Java7/2232/

All tests passed

Build Log:
[...truncated 32769 lines...]
-check-forbidden-all:
[forbidden-apis] Reading bundled API signatures: jdk-unsafe-1.7
[forbidden-apis] Reading bundled API signatures: jdk-deprecated-1.7
[forbidden-apis] Reading bundled API signatures: commons-io-unsafe-2.4
[forbidden-apis] Reading API signatures: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/lucene/tools/forbiddenApis/base.txt
[forbidden-apis] Reading API signatures: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/lucene/tools/forbiddenApis/servlet-api.txt
[forbidden-apis] Loading classes to check...
[forbidden-apis] Scanning for API signatures and dependencies...
[forbidden-apis] Forbidden method invocation: java.io.InputStreamReader#<init>(java.io.InputStream) [Uses default charset]
[forbidden-apis]   in org.apache.solr.core.SolrConfig (SolrConfig.java:356)
[forbidden-apis] Scanned 2289 (and 1562 related) class file(s) for forbidden API invocations (in 2.49s), 1 error(s).

BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/build.xml:525: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/build.xml:86: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build.xml:271: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/common-build.xml:491: Check for forbidden API calls failed, see log.

Total time: 99 minutes 15 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-Tests-5.x-Java7 #2226
Archived 1 artifacts
Archive block size is 32768
Received 0 blocks and 464 bytes
Compression is 0.0%
Took 1.4 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-5.x-Java7 - Build # 2231 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-5.x-Java7/2231/

1 tests failed.
REGRESSION:  org.apache.solr.cloud.DistribDocExpirationUpdateProcessorTest.testDistribSearch

Error Message:
There are still nodes recoverying - waited for 30 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 30 seconds
	at __randomizedtesting.SeedInfo.seed([AB214BAFB9E96B29:2AC7C5B7CEB60B15]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:178)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:840)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1459)
	at org.apache.solr.cloud.DistribDocExpirationUpdateProcessorTest.doTest(DistribDocExpirationUpdateProcessorTest.java:79)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11535 lines...]
   [junit4] Suite: org.apache.solr.cloud.DistribDocExpirationUpdateProcessorTest
   [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/init-core-data-001
   [junit4]   2> 1888711 T5452 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 1888711 T5452 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1888715 T5452 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 1888716 T5452 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1888716 T5453 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1888816 T5452 oasc.ZkTestServer.run start zk server on port:33406
   [junit4]   2> 1888817 T5452 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1888818 T5452 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1888821 T5460 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@352aace0 name:ZooKeeperConnection Watcher:127.0.0.1:33406 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1888821 T5452 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1888822 T5452 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1888822 T5452 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1888824 T5452 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1888825 T5452 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1888826 T5463 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@aa321eb name:ZooKeeperConnection Watcher:127.0.0.1:33406/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1888826 T5452 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1888827 T5452 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1888827 T5452 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1888829 T5452 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1888830 T5452 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1888831 T5452 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1888833 T5452 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/solrconfig-doc-expire-update-processor.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1888833 T5452 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1888835 T5452 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 1888836 T5452 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1888838 T5452 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1888838 T5452 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1888840 T5452 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1888840 T5452 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1888841 T5452 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1888842 T5452 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1888843 T5452 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1888844 T5452 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1888845 T5452 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1888846 T5452 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1888847 T5452 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1888847 T5452 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1888849 T5452 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1888849 T5452 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1888851 T5452 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1888851 T5452 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1888853 T5452 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1888853 T5452 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1888855 T5452 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1888855 T5452 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1888857 T5466 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4d77c790 name:ZooKeeperConnection Watcher:127.0.0.1:33406/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1888857 T5452 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1888857 T5452 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1890207 T5452 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1890213 T5452 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1890218 T5452 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:33418
   [junit4]   2> 1890221 T5452 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1890222 T5452 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1890222 T5452 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-002
   [junit4]   2> 1890222 T5452 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-002/'
   [junit4]   2> 1890286 T5452 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-002/solr.xml
   [junit4]   2> 1890391 T5452 oasc.CoreContainer.<init> New CoreContainer 1215268753
   [junit4]   2> 1890391 T5452 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-002/]
   [junit4]   2> 1890393 T5452 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1890393 T5452 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1890393 T5452 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1890393 T5452 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1890393 T5452 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1890394 T5452 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1890394 T5452 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1890394 T5452 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1890394 T5452 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1890395 T5452 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1890397 T5452 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1890399 T5452 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1890399 T5452 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1890399 T5452 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1890399 T5452 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:33406/solr
   [junit4]   2> 1890400 T5452 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1890400 T5452 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1890401 T5452 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1890402 T5478 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@46e63c71 name:ZooKeeperConnection Watcher:127.0.0.1:33406 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1890403 T5452 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1890404 T5452 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1890406 T5452 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1890407 T5481 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e2c1525 name:ZooKeeperConnection Watcher:127.0.0.1:33406/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1890407 T5452 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1890408 T5452 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1890410 T5452 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1890412 T5452 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1890414 T5452 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1890415 T5452 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1890417 T5452 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1890418 T5452 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33418_
   [junit4]   2> 1890419 T5452 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33418_
   [junit4]   2> 1890421 T5452 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1890422 T5452 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1890423 T5452 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1890424 T5452 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:33418_
   [junit4]   2> 1890425 T5452 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1890426 T5452 oasc.Overseer.start Overseer (id=92857431635394564-127.0.0.1:33418_-n_0000000000) starting
   [junit4]   2> 1890427 T5452 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1890433 T5452 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1890433 T5483 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1890434 T5452 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1890436 T5452 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1890437 T5452 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1890439 T5482 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1890443 T5485 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 1890443 T5485 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1890444 T5485 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1890444 T5481 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1890446 T5482 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:33418",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "node_name":"127.0.0.1:33418_",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "roles":null}
   [junit4]   2> 1890446 T5482 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 1890446 T5482 oasc.Overseer$ClusterStateUpdater.createCollection state version control_collection 1
   [junit4]   2> 1890446 T5482 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1890448 T5481 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1891444 T5485 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1891445 T5485 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 1891445 T5485 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1891446 T5485 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1891446 T5485 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1891447 T5485 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-002/collection1/'
   [junit4]   2> 1891450 T5485 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 1891451 T5485 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 1891452 T5485 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 1891490 T5485 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 1891493 T5485 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 1891526 T5485 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1891569 T5485 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1891570 T5485 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1891580 T5485 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1891881 T5485 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1891883 T5485 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1891884 T5485 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1891887 T5485 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1891902 T5485 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1891902 T5485 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1891902 T5485 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-002/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-001/control/data/
   [junit4]   2> 1891902 T5485 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 1891903 T5485 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-001/control/data
   [junit4]   2> 1891903 T5485 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-001/control/data/index/
   [junit4]   2> 1891903 T5485 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1891903 T5485 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-001/control/data/index
   [junit4]   2> 1891904 T5485 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=47, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=40.9501953125, floorSegmentMB=0.525390625, forceMergeDeletesPctAllowed=13.897808801016948, segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1891904 T5485 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@30ec5daa lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6df95cfd),segFN=segments_1,generation=1}
   [junit4]   2> 1891905 T5485 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1891905 T5485 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "convert-ttl-defaults"
   [junit4]   2> 1891966 T5485 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "convert-ttl-field"
   [junit4]   2> 1892036 T5485 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "convert-ttl-param"
   [junit4]   2> 1892037 T5485 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "convert-ttl-field-with-param-default"
   [junit4]   2> 1892037 T5485 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "scheduled-delete" (default)
   [junit4]   2> 1892102 T5485 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "scheduled-delete" (default)
   [junit4]   2> 1892102 T5485 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 1892103 T5485 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1892103 T5485 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1892103 T5485 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1892103 T5485 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1892103 T5485 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1892103 T5485 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1892104 T5485 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 1892104 T5485 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1892151 T5485 oasc.RequestHandlers.initHandlersFromConfig created /admin/: solr.admin.AdminHandlers
   [junit4]   2> 1892177 T5485 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1892180 T5485 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1892182 T5485 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1892185 T5485 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1892195 T5485 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1892196 T5485 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1892196 T5485 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=22, maxMergeAtOnceExplicit=46, maxMergedSegmentMB=51.916015625, floorSegmentMB=0.2138671875, forceMergeDeletesPctAllowed=22.761472483349653, segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4179582323608434
   [junit4]   2> 1892197 T5485 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@30ec5daa lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6df95cfd),segFN=segments_1,generation=1}
   [junit4]   2> 1892197 T5485 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1892198 T5485 oass.SolrIndexSearcher.<init> Opening Searcher@14d4d8b3[collection1] main
   [junit4]   2> 1892198 T5485 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1892199 T5485 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1892200 T5485 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1892200 T5485 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1892200 T5485 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1892201 T5485 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1892201 T5485 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1892202 T5485 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1892202 T5485 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 1892205 T5485 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1892205 T5485 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1892206 T5485 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1892230 T5485 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1892230 T5486 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@14d4d8b3[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1892231 T5490 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:33418 collection:control_collection shard:shard1
   [junit4]   2> 1892231 T5452 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3
   [junit4]   2> 1892232 T5452 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1892232 T5490 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1892238 T5490 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1892239 T5452 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1892240 T5490 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1892240 T5481 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1892240 T5452 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1892240 T5490 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C4613 name=collection1 org.apache.solr.core.SolrCore@125a9996 url=https://127.0.0.1:33418/collection1 node=127.0.0.1:33418_ C4613_STATE=coll:control_collection core:collection1 props:{base_url=https://127.0.0.1:33418, state=down, core=collection1, node_name=127.0.0.1:33418_}
   [junit4]   2> 1892241 T5490 C4613 P33418 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:33418/collection1/
   [junit4]   2> 1892242 T5490 C4613 P33418 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1892242 T5490 C4613 P33418 oasc.SyncStrategy.syncToMe https://127.0.0.1:33418/collection1/ has no replicas
   [junit4]   2> 1892242 T5490 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:33418/collection1/ shard1
   [junit4]   2> 1892243 T5493 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d893e0f name:ZooKeeperConnection Watcher:127.0.0.1:33406/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1892243 T5490 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1892243 T5452 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1892244 T5482 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection"}
   [junit4]   2> 1892244 T5452 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1892251 T5452 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1892253 T5481 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1892253 T5493 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 1892256 T5452 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1892257 T5452 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 1892257 T5452 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1892258 T5452 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1892259 T5482 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:33418",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"}
   [junit4]   2> 1892260 T5496 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3df5e449 name:ZooKeeperConnection Watcher:127.0.0.1:33406/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1892260 T5452 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1892261 T5452 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1892261 T5481 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1892262 T5481 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1892262 T5493 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1892267 T5482 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "name":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "stateFormat":2}
   [junit4]   2> 1892267 T5482 oasc.Overseer$ClusterStateUpdater.buildCollection Building a new collection: collection1
   [junit4]   2> 1892267 T5482 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 1892268 T5482 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2
   [junit4]   2> 1892269 T5481 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1892270 T5482 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/collection1/state.json
   [junit4]   2> 1892271 T5481 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1892271 T5493 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1892306 T5490 oasc.ZkController.register We are https://127.0.0.1:33418/collection1/ and leader is https://127.0.0.1:33418/collection1/
   [junit4]   2> 1892307 T5490 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:33418
   [junit4]   2> 1892307 T5490 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1892307 T5490 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 1892307 T5490 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1892309 T5481 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1892311 T5482 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:33418",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "node_name":"127.0.0.1:33418_",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "roles":null}
   [junit4]   2> 1892414 T5493 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1892414 T5481 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1893662 T5452 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 1893664 T5452 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1893670 T5452 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1893672 T5452 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:33695
   [junit4]   2> 1893676 T5452 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1893676 T5452 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1893676 T5452 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-003
   [junit4]   2> 1893677 T5452 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-003/'
   [junit4]   2> 1893725 T5452 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-003/solr.xml
   [junit4]   2> 1893810 T5452 oasc.CoreContainer.<init> New CoreContainer 134204882
   [junit4]   2> 1893810 T5452 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-003/]
   [junit4]   2> 1893811 T5452 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1893812 T5452 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1893812 T5452 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1893812 T5452 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1893813 T5452 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1893813 T5452 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1893813 T5452 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1893814 T5452 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1893814 T5452 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1893814 T5452 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1893818 T5452 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1893820 T5452 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1893821 T5452 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1893821 T5452 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1893822 T5452 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:33406/solr
   [junit4]   2> 1893822 T5452 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1893822 T5452 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1893823 T5452 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1893825 T5508 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@361a4c72 name:ZooKeeperConnection Watcher:127.0.0.1:33406 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1893826 T5452 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1893827 T5452 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1893828 T5452 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1893830 T5511 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@27f82c3f name:ZooKeeperConnection Watcher:127.0.0.1:33406/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1893830 T5452 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1893836 T5452 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1894840 T5452 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33695_
   [junit4]   2> 1894841 T5452 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33695_
   [junit4]   2> 1894844 T5452 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1894852 T5512 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1894852 T5512 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1894854 T5481 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1894854 T5512 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 1894854 T5512 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 1894855 T5512 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 1894856 T5512 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 0 
   [junit4]   2> 1894856 T5512 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1894859 T5482 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:33695",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "node_name":"127.0.0.1:33695_",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "roles":null}
   [junit4]   2> 1894861 T5482 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 1894862 T5482 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1894865 T5481 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1894865 T5482 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 1894866 T5511 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1894867 T5511 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 1 
   [junit4]   2>  C4613_STATE=coll:control_collection core:collection1 props:{base_url=https://127.0.0.1:33418, state=active, core=collection1, node_name=127.0.0.1:33418_, leader=true}
   [junit4]   2> 1895230 T5487 C4613 P33418 oasup.DocExpirationUpdateProcessorFactory$DeleteExpiredDocsRunnable.run Begining periodic deletion of expired docs
   [junit4]   2> 1895234 T5487 C4613 P33418 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
   [junit4]   2> 1895235 T5487 C4613 P33418 oasc.SolrCore.openNewSearcher SolrIndexSearcher has not changed - not re-opening: org.apache.solr.search.SolrIndexSearcher
   [junit4]   2> 1895235 T5487 C4613 P33418 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 1895235 T5487 C4613 P33418 oasup.LogUpdateProcessor.finish [collection1] {deleteByQuery={!cache=false}eXpField_tdt:[* TO 2014-11-25T05:06:08.72Z] (-1485718912993591296),commit=} 0 5
   [junit4]   2> 1895235 T5487 C4613 P33418 oasup.DocExpirationUpdateProcessorFactory$DeleteExpiredDocsRunnable.run Finished periodic deletion of expired docs
   [junit4]   2> 1895856 T5512 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1895857 T5512 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1895857 T5512 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1895858 T5512 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1895858 T5512 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1895858 T5512 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-003/collection1/'
   [junit4]   2> 1895860 T5512 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 1895861 T5512 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 1895862 T5512 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 1895895 T5512 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 1895897 T5512 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 1895930 T5512 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1895972 T5512 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1895974 T5512 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1895983 T5512 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1896324 T5512 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1896327 T5512 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1896329 T5512 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1896333 T5512 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1896351 T5512 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1896351 T5512 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1896351 T5512 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-003/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-001/jetty1/
   [junit4]   2> 1896352 T5512 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 1896352 T5512 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-001/jetty1
   [junit4]   2> 1896353 T5512 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-001/jetty1/index/
   [junit4]   2> 1896353 T5512 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 1896354 T5512 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-001/jetty1/index
   [junit4]   2> 1896354 T5512 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=47, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=40.9501953125, floorSegmentMB=0.525390625, forceMergeDeletesPctAllowed=13.897808801016948, segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1896355 T5512 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@6b1b1e96 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@417540c9),segFN=segments_1,generation=1}
   [junit4]   2> 1896355 T5512 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1896356 T5512 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "convert-ttl-defaults"
   [junit4]   2> 1896357 T5512 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "convert-ttl-field"
   [junit4]   2> 1896357 T5512 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "convert-ttl-param"
   [junit4]   2> 1896357 T5512 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "convert-ttl-field-with-param-default"
   [junit4]   2> 1896358 T5512 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "scheduled-delete" (default)
   [junit4]   2> 1896358 T5512 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "scheduled-delete" (default)
   [junit4]   2> 1896358 T5512 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 1896359 T5512 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1896359 T5512 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1896359 T5512 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1896359 T5512 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1896360 T5512 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1896360 T5512 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1896360 T5512 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 1896360 T5512 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1896361 T5512 oasc.RequestHandlers.initHandlersFromConfig created /admin/: solr.admin.AdminHandlers
   [junit4]   2> 1896379 T5512 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1896381 T5512 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1896383 T5512 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1896385 T5512 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1896389 T5512 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1896389 T5512 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1896390 T5512 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=22, maxMergeAtOnceExplicit=46, maxMergedSegmentMB=51.916015625, floorSegmentMB=0.2138671875, forceMergeDeletesPctAllowed=22.761472483349653, segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4179582323608434
   [junit4]   2> 1896390 T5512 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@6b1b1e96 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@417540c9),segFN=segments_1,generation=1}
   [junit4]   2> 1896391 T5512 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1896391 T5512 oass.SolrIndexSearcher.<init> Opening Searcher@43184fb8[collection1] main
   [junit4]   2> 1896391 T5512 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1896392 T5512 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1896392 T5512 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1896393 T5512 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1896393 T5512 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1896393 T5512 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1896394 T5512 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1896394 T5512 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1896394 T5512 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 1896394 T5512 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1896396 T5513 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@43184fb8[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1896396 T5512 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1896397 T5517 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:33695 collection:collection1 shard:shard2
   [junit4]   2> 1896398 T5452 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3
   [junit4]   2> 1896398 T5452 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1896398 T5517 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 1896402 T5517 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 1896404 T5481 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1896404 T5517 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1896404 T5517 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C4614 name=collection1 org.apache.solr.core.SolrCore@74b70075 url=https://127.0.0.1:33695/collection1 node=127.0.0.1:33695_ C4614_STATE=coll:collection1 core:collection1 props:{base_url=https://127.0.0.1:33695, state=down, core=collection1, node_name=127.0.0.1:33695_}
   [junit4]   2> 1896405 T5517 C4614 P33695 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:33695/collection1/
   [junit4]   2> 1896405 T5517 C4614 P33695 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1896405 T5517 C4614 P33695 oasc.SyncStrategy.syncToMe https://127.0.0.1:33695/collection1/ has no replicas
   [junit4]   2> 1896405 T5517 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:33695/collection1/ shard2
   [junit4]   2> 1896406 T5517 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 1896407 T5482 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1"}
   [junit4]   2> 1896410 T5482 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 1896410 T5511 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1896411 T5511 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 2 
   [junit4]   2> 1896414 T5482 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:33695",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"}
   [junit4]   2> 1896416 T5481 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1896416 T5482 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 1896417 T5511 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1896417 T5511 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 
   [junit4]   2> 1896460 T5517 oasc.ZkController.register We are https://127.0.0.1:33695/collection1/ and leader is https://127.0.0.1:33695/collection1/
   [junit4]   2> 1896461 T5517 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:33695
   [junit4]   2> 1896461 T5517 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1896461 T5517 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 1896461 T5517 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1896463 T5481 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1896465 T5517 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 
   [junit4]   2> 1896465 T5482 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:33695",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "node_name":"127.0.0.1:33695_",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "roles":null}
   [junit4]   2> 1896469 T5482 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 1896469 T5511 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1896470 T5511 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 1897581 T5452 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 1897582 T5452 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1897586 T5452 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1897587 T5452 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:34006
   [junit4]   2> 1897590 T5452 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1897590 T5452 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1897590 T5452 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-004
   [junit4]   2> 1897590 T5452 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-004/'
   [junit4]   2> 1897629 T5452 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-004/solr.xml
   [junit4]   2> 1897704 T5452 oasc.CoreContainer.<init> New CoreContainer 179146714
   [junit4]   2> 1897704 T5452 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-004/]
   [junit4]   2> 1897705 T5452 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1897705 T5452 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1897706 T5452 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1897706 T5452 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1897706 T5452 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1897706 T5452 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1897707 T5452 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1897707 T5452 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1897707 T5452 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1897707 T5452 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1897709 T5452 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1897711 T5452 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1897711 T5452 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1897711 T5452 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1897712 T5452 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:33406/solr
   [junit4]   2> 1897712 T5452 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1897712 T5452 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1897713 T5452 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1897715 T5529 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f244172 name:ZooKeeperConnection Watcher:127.0.0.1:33406 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1897715 T5452 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1897716 T5452 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1897718 T5452 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1897719 T5532 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7dc55e65 name:ZooKeeperConnection Watcher:127.0.0.1:33406/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1897719 T5452 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1897724 T5452 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C4615 name=collection1 org.apache.solr.core.SolrCore@125a9996 url=https://127.0.0.1:33418/collection1 node=127.0.0.1:33418_ C4615_STATE=coll:control_collection core:collection1 props:{base_url=https://127.0.0.1:33418, state=active, core=collection1, node_name=127.0.0.1:33418_, leader=true}
   [junit4]   2> 1898231 T5487 C4615 P33418 oasup.DocExpirationUpdateProcessorFactory$DeleteExpiredDocsRunnable.run Begining periodic deletion of expired docs
   [junit4]   2> 1898232 T5487 C4615 P33418 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
   [junit4]   2> 1898232 T5487 C4615 P33418 oasc.SolrCore.openNewSearcher SolrIndexSearcher has not changed - not re-opening: org.apache.solr.search.SolrIndexSearcher
   [junit4]   2> 1898232 T5487 C4615 P33418 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 1898233 T5487 C4615 P33418 oasup.LogUpdateProcessor.finish [collection1] {deleteByQuery={!cache=false}eXpField_tdt:[* TO 2014-11-25T05:06:11.72Z] (-1485718916139319296),commit=} 0 3
   [junit4]   2> 1898233 T5487 C4615 P33418 oasup.DocExpirationUpdateProcessorFactory$DeleteExpiredDocsRunnable.run Finished periodic deletion of expired docs
   [junit4]   2> 1898728 T5452 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34006_
   [junit4]   2> 1898729 T5452 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:34006_
   [junit4]   2> 1898733 T5452 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1898741 T5533 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1898741 T5533 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1898743 T5481 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1898743 T5533 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 1898743 T5533 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 1898744 T5533 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 1898745 T5533 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 1898745 T5533 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1898745 T5482 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:34006",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "node_name":"127.0.0.1:34006_",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "roles":null}
   [junit4]   2> 1898748 T5482 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 1898749 T5482 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1898753 T5482 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 1898754 T5532 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1898754 T5511 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1898754 T5532 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> 1898755 T5511 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> ASYNC  NEW_CORE C4616 name=collection1 org.apache.solr.core.SolrCore@74b70075 url=https://127.0.0.1:33695/collection1 node=127.0.0.1:33695_ C4616_STATE=coll:collection1 core:collection1 props:{base_url=https://127.0.0.1:33695, state=active, core=collection1, node_name=127.0.0.1:33695_, leader=true}
   [junit4]   2> 1899396 T5514 C4616 P33695 oasup.DocExpirationUpdateProcessorFactory.iAmInChargeOfPeriodicDeletes WARN Slice in charge of periodic deletes for collection1 does not currently have a leader
   [junit4]   2> 1899747 T5533 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1899747 T5533 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1899748 T5533 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1899748 T5533 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1899749 T5533 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1899749 T5533 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-004/collection1/'
   [junit4]   2> 1899752 T5533 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 1899753 T5533 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 1899754 T5533 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 1899795 T5533 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 1899797 T5533 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 1899831 T5533 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1899874 T5533 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1899875 T5533 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1899884 T5533 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1900229 T5533 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1900232 T5533 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1900233 T5533 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1900238 T5533 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1900254 T5533 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1900254 T5533 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1900255 T5533 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-004/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-001/jetty2/
   [junit4]   2> 1900255 T5533 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 1900256 T5533 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-001/jetty2
   [junit4]   2> 1900256 T5533 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-001/jetty2/index/
   [junit4]   2> 1900256 T5533 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 1900257 T5533 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-001/jetty2/index
   [junit4]   2> 1900257 T5533 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=47, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=40.9501953125, floorSegmentMB=0.525390625, forceMergeDeletesPctAllowed=13.897808801016948, segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1900258 T5533 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@209b270d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2a20d734),segFN=segments_1,generation=1}
   [junit4]   2> 1900259 T5533 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1900260 T5533 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "convert-ttl-defaults"
   [junit4]   2> 1900260 T5533 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "convert-ttl-field"
   [junit4]   2> 1900260 T5533 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "convert-ttl-param"
   [junit4]   2> 1900261 T5533 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "convert-ttl-field-with-param-default"
   [junit4]   2> 1900261 T5533 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "scheduled-delete" (default)
   [junit4]   2> 1900261 T5533 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "scheduled-delete" (default)
   [junit4]   2> 1900262 T5533 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 1900262 T5533 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1900262 T5533 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1900263 T5533 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1900263 T5533 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1900263 T5533 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1900263 T5533 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1900263 T5533 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 1900264 T5533 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1900264 T5533 oasc.RequestHandlers.initHandlersFromConfig created /admin/: solr.admin.AdminHandlers
   [junit4]   2> 1900290 T5533 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1900292 T5533 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1900295 T5533 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1900298 T5533 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1900313 T5533 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1900314 T5533 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1900314 T5533 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=22, maxMergeAtOnceExplicit=46, maxMergedSegmentMB=51.916015625, floorSegmentMB=0.2138671875, forceMergeDeletesPctAllowed=22.761472483349653, segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4179582323608434
   [junit4]   2> 1900315 T5533 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@209b270d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2a20d734),segFN=segments_1,generation=1}
   [junit4]   2> 1900315 T5533 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1900315 T5533 oass.SolrIndexSearcher.<init> Opening Searcher@4c5732e3[collection1] main
   [junit4]   2> 1900315 T5533 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1900316 T5533 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1900316 T5533 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1900317 T5533 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1900317 T5533 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1900317 T5533 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1900318 T5533 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1900318 T5533 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1900318 T5533 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 1900318 T5533 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1900320 T5533 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1900320 T5534 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4c5732e3[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1900320 T5538 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:34006 collection:collection1 shard:shard1
   [junit4]   2> 1900321 T5452 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3
   [junit4]   2> 1900321 T5538 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1900321 T5452 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1900325 T5538 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1900327 T5481 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1900327 T5538 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1900328 T5538 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C4617 name=collection1 org.apache.solr.core.SolrCore@761e8548 url=https://127.0.0.1:34006/collection1 node=127.0.0.1:34006_ C4617_STATE=coll:collection1 core:collection1 props:{base_url=https://127.0.0.1:34006, state=down, core=collection1, node_name=127.0.0.1:34006_}
   [junit4]   2> 1900328 T5538 C4617 P34006 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:34006/collection1/
   [junit4]   2> 1900328 T5538 C4617 P34006 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1900328 T5538 C4617 P34006 oasc.SyncStrategy.syncToMe https://127.0.0.1:34006/collection1/ has no replicas
   [junit4]   2> 1900328 T5538 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:34006/collection1/ shard1
   [junit4]   2> 1900328 T5538 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1900330 T5482 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1"}
   [junit4]   2> 1900333 T5482 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 1900333 T5532 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1900333 T5511 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1900334 T5532 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 6 
   [junit4]   2> 1900334 T5511 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 6 
   [junit4]   2> 1900338 T5482 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:34006",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"}
   [junit4]   2> 1900341 T5481 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1900341 T5482 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 1900342 T5511 oascc.

[...truncated too long message...]

e.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:357)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:267)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1966269 T5452 oasc.Overseer.close Overseer (id=92857431635394574-127.0.0.1:34345_-n_0000000004) closing
   [junit4]   2> 1966269 T6492 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1966269 T6503 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 51 
   [junit4]   2> 1966271 T6492 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-001/jetty4/index [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-001/jetty4/index;done=true>>]
   [junit4]   2> 1966271 T5588 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1966271 T6504 oasc.Overseer$ClusterStateUpdater.run WARN Solr cannot talk to ZK, exiting Overseer main queue loop org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /live_nodes
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:331)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:328)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:328)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:509)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:255)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:292)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1966271 T6492 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-001/jetty4/index
   [junit4]   2> 1966272 T6504 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:34345_
   [junit4]   2> 1966273 T6492 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 1 directories currently being tracked
   [junit4]   2> 1966274 T6492 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-001/jetty4 [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-001/jetty4;done=false>>]
   [junit4]   2> 1966274 T6492 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001/tempDir-001/jetty4
   [junit4]   2> 1966276 T5452 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 1968069 T5452 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 1968071 T5452 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:33406 33406
   [junit4]   2> 1969226 T5453 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:33406 33406
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DistribDocExpirationUpdateProcessorTest -Dtests.method=testDistribSearch -Dtests.seed=AB214BAFB9E96B29 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=en_CA -Dtests.timezone=Etc/GMT-13 -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 80.5s J3 | DistribDocExpirationUpdateProcessorTest.testDistribSearch <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: There are still nodes recoverying - waited for 30 seconds
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([AB214BAFB9E96B29:2AC7C5B7CEB60B15]:0)
   [junit4]    > 	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:178)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:840)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1459)
   [junit4]    > 	at org.apache.solr.cloud.DistribDocExpirationUpdateProcessorTest.doTest(DistribDocExpirationUpdateProcessorTest.java:79)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1969249 T5452 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J3/temp/solr.cloud.DistribDocExpirationUpdateProcessorTest-AB214BAFB9E96B29-001
   [junit4]   2> NOTE: test params are: codec=HighCompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=395), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=395)), sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=en_CA, timezone=Etc/GMT-13
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=210136800,total=360710144
   [junit4]   2> NOTE: All tests run in this JVM: [TestOmitPositions, TestLMDirichletSimilarityFactory, RemoteQueryErrorTest, FieldAnalysisRequestHandlerTest, TestSolrQueryParserDefaultOperatorResource, TestDistribDocBasedVersion, TermsComponentTest, TestUpdate, TriLevelCompositeIdRoutingTest, TestFieldResource, TestLMJelinekMercerSimilarityFactory, TestSuggestSpellingConverter, TestIntervalFaceting, TestDynamicFieldResource, TestManagedSchemaDynamicFieldResource, TestWriterPerf, TestRestManager, XsltUpdateRequestHandlerTest, TestRequestStatusCollectionAPI, DebugComponentTest, TestSolrCoreProperties, TestAddFieldRealTimeGet, DistributedQueryComponentCustomSortTest, IndexSchemaTest, CloudMLTQParserTest, AnalyticsMergeStrategyTest, TestExtendedDismaxParser, DocValuesMultiTest, TestLazyCores, TestSearcherReuse, HdfsCollectionsAPIDistributedZkTest, OverseerCollectionProcessorTest, SuggesterFSTTest, StatelessScriptUpdateProcessorFactoryTest, TestRangeQuery, WordBreakSolrSpellCheckerTest, TestStressRecovery, UUIDFieldTest, TestPartialUpdateDeduplication, TestRandomFaceting, DocumentBuilderTest, TestManagedResource, TestRandomMergePolicy, TestPivotHelperCode, OutputWriterTest, TestCollationField, DateFieldTest, BasicZkTest, AtomicUpdatesTest, TestRealTimeGet, BasicDistributedZkTest, ZkSolrClientTest, TestCloudManagedSchemaConcurrent, PrimitiveFieldTypeTest, TestRTGBase, BasicDistributedZk2Test, TestCloudSchemaless, CustomCollectionTest, DistribDocExpirationUpdateProcessorTest]
   [junit4] Completed on J3 in 80.57s, 1 test, 1 failure <<< FAILURES!

[...truncated 756 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/build.xml:525: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/build.xml:473: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/common-build.xml:508: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/lucene/common-build.xml:1363: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/lucene/common-build.xml:970: There were test failures: 449 suites, 1825 tests, 1 failure, 59 ignored (22 assumptions)

Total time: 68 minutes 42 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-Tests-5.x-Java7 #2226
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 183946969 bytes
Compression is 0.0%
Took 32 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-5.x-Java7 - Build # 2230 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-5.x-Java7/2230/

1 tests failed.
REGRESSION:  org.apache.solr.client.solrj.TestLBHttpSolrServer.testReliability

Error Message:
No live SolrServers available to handle this request

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request
	at __randomizedtesting.SeedInfo.seed([2B5CEEF309D6B573:EA9433B5A8B064DA]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:539)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.client.solrj.TestLBHttpSolrServer.testReliability(TestLBHttpSolrServer.java:223)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:45317/solr
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:579)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:492)
	... 43 more
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:152)
	at java.net.SocketInputStream.read(SocketInputStream.java:122)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:465)
	... 46 more




Build Log:
[...truncated 12695 lines...]
   [junit4] Suite: org.apache.solr.client.solrj.TestLBHttpSolrServer
   [junit4]   2> SLF4J: Class path contains multiple SLF4J bindings.
   [junit4]   2> SLF4J: Found binding in [jar:file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/solrj/test-lib/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
   [junit4]   2> SLF4J: Found binding in [jar:file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/lib/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
   [junit4]   2> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
   [junit4]   2> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
   [junit4]   2> log4j:WARN No such property [conversionPattern] in org.apache.solr.util.SolrLogLayout.
   [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/init-core-data-001
   [junit4]   2> 5466 T11 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 5790 T11 oas.SolrTestCaseJ4.setUp ###Starting testReliability
   [junit4]   2> 8141 T11 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 8698 T11 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:21977
   [junit4]   2> 8857 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 8993 T11 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 8995 T11 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001
   [junit4]   2> 8996 T11 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/'
   [junit4]   2> 10120 T11 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/solr.xml
   [junit4]   2> 10640 T11 oasc.CoreContainer.<init> New CoreContainer 1748742355
   [junit4]   2> 10654 T11 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/]
   [junit4]   2> 11027 T11 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 11028 T11 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 11029 T11 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 11031 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 11032 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 11036 T11 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 11038 T11 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 11039 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 11040 T11 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 11041 T11 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 11786 T11 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 11835 T11 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 11867 T11 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 11927 T11 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 13138 T23 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/'
   [junit4]   2> 13581 T23 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 13995 T23 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 14100 T23 oass.IndexSchema.readSchema Reading Solr Schema from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/conf/schema.xml
   [junit4]   2> 14181 T23 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 14784 T23 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 14810 T23 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/
   [junit4]   2> 15205 T23 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockFSDirectoryFactory
   [junit4]   2> 15264 T23 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/data/
   [junit4]   2> 15266 T23 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 15479 T23 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/data
   [junit4]   2> 15483 T23 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/data/index/
   [junit4]   2> 15486 T23 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 15531 T23 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/data/index
   [junit4]   2> 15894 T23 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@17aae3f3),segFN=segments_1,generation=1}
   [junit4]   2> 15900 T23 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 18575 T23 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 18768 T23 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 18769 T23 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 18770 T23 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 18772 T23 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 18858 T23 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 18859 T23 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 18860 T23 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 18870 T23 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 18907 T23 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 19024 T23 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 19296 T23 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 19335 T23 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 19339 T23 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 19342 T23 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 19486 T23 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 19488 T23 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 19531 T23 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@17aae3f3),segFN=segments_1,generation=1}
   [junit4]   2> 19533 T23 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 19804 T23 oass.SolrIndexSearcher.<init> Opening Searcher@5bd904[collection1] main
   [junit4]   2> 19888 T23 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/conf
   [junit4]   2> 19889 T23 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/conf}
   [junit4]   2> 20089 T23 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/conf
   [junit4]   2> 20091 T23 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 20114 T23 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using file:dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/conf
   [junit4]   2> 20115 T23 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 20371 T23 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 20372 T23 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 20386 T24 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5bd904[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 20395 T11 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0
   [junit4]   2> 20396 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> ASYNC  NEW_CORE C0 name=collection1 org.apache.solr.core.SolrCore@5b2e5ec7
   [junit4]   2> 24385 T19 C0 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]} 0 1899
   [junit4]   2> 24799 T19 C0 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 25886 T19 C0 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@17aae3f3),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@17aae3f3),segFN=segments_2,generation=2}
   [junit4]   2> 25889 T19 C0 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 26286 T19 C0 oass.SolrIndexSearcher.<init> Opening Searcher@49a27fbe[collection1] main
   [junit4]   2> 26296 T19 C0 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 26310 T24 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@49a27fbe[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(5.0.0):C10)))}
   [junit4]   2> 26312 T19 C0 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&waitSearcher=true&commit=true&softCommit=false&wt=javabin} {commit=} 0 1513
   [junit4]   2> 26337 T11 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 26340 T11 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:58148
   [junit4]   2> 26342 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 26343 T11 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 26344 T11 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001
   [junit4]   2> 26345 T11 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/'
   [junit4]   2> 26410 T11 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/solr.xml
   [junit4]   2> 26550 T11 oasc.CoreContainer.<init> New CoreContainer 1378970726
   [junit4]   2> 26551 T11 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/]
   [junit4]   2> 26554 T11 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 26554 T11 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 26555 T11 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 26556 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 26557 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 26558 T11 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 26559 T11 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 26560 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 26561 T11 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 26561 T11 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 26563 T11 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 26564 T11 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 26565 T11 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 26566 T11 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 26575 T35 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/collection1/'
   [junit4]   2> 26641 T35 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 26716 T35 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 26718 T35 oass.IndexSchema.readSchema Reading Solr Schema from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/collection1/conf/schema.xml
   [junit4]   2> 26728 T35 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 26752 T35 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 26754 T35 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/collection1/
   [junit4]   2> 26755 T35 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockFSDirectoryFactory
   [junit4]   2> 26756 T35 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/collection1/data/
   [junit4]   2> 26756 T35 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 26758 T35 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/collection1/data
   [junit4]   2> 26760 T35 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/collection1/data/index/
   [junit4]   2> 26761 T35 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 26763 T35 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/collection1/data/index
   [junit4]   2> 26771 T35 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/collection1/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@17aae3f3),segFN=segments_1,generation=1}
   [junit4]   2> 26772 T35 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 26775 T35 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 26775 T35 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 26776 T35 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 26777 T35 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 26777 T35 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 26778 T35 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 26778 T35 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 26779 T35 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 26779 T35 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 26780 T35 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 26781 T35 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 26801 T35 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 26804 T35 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 26806 T35 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 26809 T35 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 26812 T35 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 26812 T35 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 26816 T35 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/collection1/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@17aae3f3),segFN=segments_1,generation=1}
   [junit4]   2> 26817 T35 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 26818 T35 oass.SolrIndexSearcher.<init> Opening Searcher@4ee8ff06[collection1] main
   [junit4]   2> 26819 T35 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/collection1/conf
   [junit4]   2> 26820 T35 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/collection1/conf}
   [junit4]   2> 26822 T35 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/collection1/conf
   [junit4]   2> 26823 T35 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 26825 T35 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using file:dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/collection1/conf
   [junit4]   2> 26825 T35 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 26826 T35 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 26827 T36 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4ee8ff06[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 26827 T35 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 26828 T11 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0
   [junit4]   2> 26829 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> ASYNC  NEW_CORE C1 name=collection1 org.apache.solr.core.SolrCore@64c88cb
   [junit4]   2> 26856 T28 C1 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]} 0 10
   [junit4]   2> 26863 T28 C1 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 26895 T28 C1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/collection1/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@17aae3f3),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/collection1/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@17aae3f3),segFN=segments_2,generation=2}
   [junit4]   2> 26896 T28 C1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 26937 T28 C1 oass.SolrIndexSearcher.<init> Opening Searcher@5a6bc081[collection1] main
   [junit4]   2> 26938 T28 C1 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 26938 T36 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5a6bc081[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(5.0.0):C10)))}
   [junit4]   2> 26940 T28 C1 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&waitSearcher=true&commit=true&softCommit=false&wt=javabin} {commit=} 0 77
   [junit4]   2> 26958 T11 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 26961 T11 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:45317
   [junit4]   2> 26963 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 26963 T11 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 26964 T11 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001
   [junit4]   2> 26965 T11 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/'
   [junit4]   2> 26983 T11 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/solr.xml
   [junit4]   2> 27082 T11 oasc.CoreContainer.<init> New CoreContainer 1113453434
   [junit4]   2> 27083 T11 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/]
   [junit4]   2> 27085 T11 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 27086 T11 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 27087 T11 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 27087 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 27088 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 27088 T11 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 27089 T11 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 27089 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 27090 T11 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 27091 T11 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 27092 T11 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 27092 T11 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 27093 T11 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 27093 T11 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 27100 T47 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/collection1/'
   [junit4]   2> 27217 T47 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 27275 T47 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 27276 T47 oass.IndexSchema.readSchema Reading Solr Schema from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/collection1/conf/schema.xml
   [junit4]   2> 27287 T47 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 27315 T47 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 27317 T47 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/collection1/
   [junit4]   2> 27317 T47 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockFSDirectoryFactory
   [junit4]   2> 27318 T47 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/collection1/data/
   [junit4]   2> 27319 T47 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 27321 T47 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/collection1/data
   [junit4]   2> 27322 T47 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/collection1/data/index/
   [junit4]   2> 27323 T47 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 27325 T47 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/collection1/data/index
   [junit4]   2> 27339 T47 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/collection1/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@17aae3f3),segFN=segments_1,generation=1}
   [junit4]   2> 27339 T47 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 27343 T47 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 27343 T47 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 27344 T47 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 27345 T47 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 27345 T47 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 27346 T47 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 27346 T47 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 27346 T47 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 27347 T47 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 27348 T47 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 27348 T47 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 27351 T47 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 27353 T47 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 27355 T47 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 27358 T47 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 27361 T47 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 27361 T47 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 27366 T47 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/collection1/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@17aae3f3),segFN=segments_1,generation=1}
   [junit4]   2> 27366 T47 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 27367 T47 oass.SolrIndexSearcher.<init> Opening Searcher@176ab7ae[collection1] main
   [junit4]   2> 27368 T47 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/collection1/conf
   [junit4]   2> 27368 T47 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/collection1/conf}
   [junit4]   2> 27369 T47 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/collection1/conf
   [junit4]   2> 27370 T47 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 27371 T47 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using file:dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/collection1/conf
   [junit4]   2> 27372 T47 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 27372 T47 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 27373 T48 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@176ab7ae[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 27373 T47 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 27374 T11 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0
   [junit4]   2> 27375 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> ASYNC  NEW_CORE C2 name=collection1 org.apache.solr.core.SolrCore@5023e78c
   [junit4]   2> 27385 T40 C2 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]} 0 6
   [junit4]   2> 27390 T40 C2 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 27416 T40 C2 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/collection1/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@17aae3f3),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/collection1/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@17aae3f3),segFN=segments_2,generation=2}
   [junit4]   2> 27417 T40 C2 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 27423 T40 C2 oass.SolrIndexSearcher.<init> Opening Searcher@75d6c6da[collection1] main
   [junit4]   2> 27424 T40 C2 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 27424 T48 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@75d6c6da[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(5.0.0):C10)))}
   [junit4]   2> 27425 T40 C2 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&waitSearcher=true&commit=true&softCommit=false&wt=javabin} {commit=} 0 35
   [junit4]   2> 27428 T11 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1378970726
   [junit4]   2> 27429 T11 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@64c88cb
   [junit4]   2> 27430 T11 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=10,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 27431 T11 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 27431 T11 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 27432 T11 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 27436 T11 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 27437 T11 oasc.CachingDirectoryFactory.close Closing MockFSDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 27438 T11 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/collection1/data [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/collection1/data;done=false>>]
   [junit4]   2> 27439 T11 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/collection1/data
   [junit4]   2> 27451 T11 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/collection1/data/index [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/collection1/data/index;done=false>>]
   [junit4]   2> 27451 T11 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/collection1/data/index
   [junit4]   2> 27657 T11 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
   [junit4]   2> 33271 T11 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1748742355
   [junit4]   2> 33271 T50 oahic.DefaultRequestDirector.tryExecute I/O exception (java.net.SocketException) caught when processing request: Connection reset
   [junit4]   2> 33273 T11 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
   [junit4]   2> 33275 T50 oahic.DefaultRequestDirector.tryExecute Retrying request
   [junit4]   2> ASYNC  NEW_CORE C3 name=collection1 org.apache.solr.core.SolrCore@5b2e5ec7
   [junit4]   2> 33371 T17 C3 oasc.SolrCore.execute [collection1] webapp=/solr path=/select params={q=*:*&version=2&wt=javabin} hits=10 status=0 QTime=869 
   [junit4]   2> ASYNC  NEW_CORE C4 name=collection1 org.apache.solr.core.SolrCore@5023e78c
   [junit4]   2> 33371 T41 C4 oasc.SolrCore.execute [collection1] webapp=/solr path=/select params={q=*:*&version=2&wt=javabin} hits=10 status=0 QTime=344 
   [junit4]   2> 33464 T17 C3 oasc.SolrException.log ERROR null:org.eclipse.jetty.io.EofException
   [junit4]   2> 		at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:142)
   [junit4]   2> 		at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:107)
   [junit4]   2> 		at org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:214)
   [junit4]   2> 		at org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:207)
   [junit4]   2> 		at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:98)
   [junit4]   2> 		at org.apache.solr.response.BinaryResponseWriter.write(BinaryResponseWriter.java:52)
   [junit4]   2> 		at org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:50)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:763)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:416)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:202)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
   [junit4]   2> 		at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:137)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
   [junit4]   2> 		at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
   [junit4]   2> 		at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
   [junit4]   2> 		at org.eclipse.jetty.server.Server.handle(Server.java:368)
   [junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
   [junit4]   2> 		at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
   [junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
   [junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
   [junit4]   2> 		at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
   [junit4]   2> 		at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
   [junit4]   2> 		at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
   [junit4]   2> 		at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
   [junit4]   2> 		at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 		at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	
   [junit4]   2> 33467 T17 C3 oasc.SolrException.log ERROR null:org.eclipse.jetty.io.EofException
   [junit4]   2> 		at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:142)
   [junit4]   2> 		at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:107)
   [junit4]   2> 		at org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:214)
   [junit4]   2> 		at org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:207)
   [junit4]   2> 		at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:98)
   [junit4]   2> 		at org.apache.solr.response.BinaryResponseWriter.write(BinaryResponseWriter.java:52)
   [junit4]   2> 		at org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:50)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:763)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:416)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:202)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
   [junit4]   2> 		at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:137)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
   [junit4]   2> 		at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
   [junit4]   2> 		at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
   [junit4]   2> 		at org.eclipse.jetty.server.Server.handle(Server.java:368)
   [junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
   [junit4]   2> 		at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
   [junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
   [junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
   [junit4]   2> 		at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
   [junit4]   2> 		at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
   [junit4]   2> 		at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
   [junit4]   2> 		at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
   [junit4]   2> 		at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 		at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	
   [junit4]   2> 33532 T21 C3 oasc.SolrCore.execute [collection1] webapp=/solr path=/select params={q=*:*&version=2&distrib=false&sort=_docid_+asc&wt=javabin&rows=0} hits=10 status=0 QTime=263 
   [junit4]   2> 33534 T21 C3 oasc.SolrException.log ERROR null:org.eclipse.jetty.io.EofException
   [junit4]   2> 		at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:142)
   [junit4]   2> 		at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:107)
   [junit4]   2> 		at org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:214)
   [junit4]   2> 		at org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:207)
   [junit4]   2> 		at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:98)
   [junit4]   2> 		at org.apache.solr.response.BinaryResponseWriter.write(BinaryResponseWriter.java:52)
   [junit4]   2> 		at org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:50)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:763)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:416)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:202)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
   [junit4]   2> 		at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:137)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
   [junit4]   2> 		at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
   [junit4]   2> 		at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
   [junit4]   2> 		at org.eclipse.jetty.server.Server.handle(Server.java:368)
   [junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
   [junit4]   2> 		at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
   [junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
   [junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
   [junit4]   2> 		at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
   [junit4]   2> 		at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
   [junit4]   2> 		at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
   [junit4]   2> 		at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
   [junit4]   2> 		at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 		at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	
   [junit4]   2> 33536 T21 C3 oasc.SolrException.log ERROR null:org.eclipse.jetty.io.EofException
   [junit4]   2> 		at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:142)
   [junit4]   2> 		at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:107)
   [junit4]   2> 		at org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:214)
   [junit4]   2> 		at org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:207)
   [junit4]   2> 		at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:98)
   [junit4]   2> 		at org.apache.solr.response.BinaryResponseWriter.write(BinaryResponseWriter.java:52)
   [junit4]   2> 		at org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:50)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:763)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:416)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:202)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
   [junit4]   2> 		at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:137)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
   [junit4]   2> 		at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
   [junit4]   2> 		at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
   [junit4]   2> 		at org.eclipse.jetty.server.Server.handle(Server.java:368)
   [junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
   [junit4]   2> 		at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
   [junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
   [junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
   [junit4]   2> 		at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
   [junit4]   2> 		at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
   [junit4]   2> 		at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
   [junit4]   2> 		at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
   [junit4]   2> 		at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 		at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	
   [junit4]   2> 33539 T21 C3 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@5b2e5ec7
   [junit4]   2> 33540 T21 C3 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=10,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 33541 T21 C3 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 33542 T21 C3 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 33542 T21 C3 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 33584 T21 C3 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 33586 T21 C3 oasc.CachingDirectoryFactory.close Closing MockFSDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 33587 T21 C3 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/data/index [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/data/index;done=false>>]
   [junit4]   2> 33588 T21 C3 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/data/index
   [junit4]   2> 33608 T21 C3 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/data [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/data;done=false>>]
   [junit4]   2> 33608 T21 C3 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/data
   [junit4]   2> 33786 T42 C4 oasc.SolrCore.execute [collection1] webapp=/solr path=/select params={q=*:*&version=2&distrib=false&sort=_docid_+asc&wt=javabin&rows=0} hits=10 status=0 QTime=1 
   [junit4]   2> 38352 T11 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1113453434
   [junit4]   2> 38353 T11 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@5023e78c
   [junit4]   2> 38354 T11 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=10,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 38355 T11 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 38355 T11 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 38356 T11 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 38365 T11 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 38366 T11 oasc.CachingDirectoryFactory.close Closing MockFSDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 38367 T11 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/collection1/data [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/collection1/data;done=false>>]
   [junit4]   2> 38367 T11 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/collection1/data
   [junit4]   2> 38369 T11 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/collection1/data/index [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/collection1/data/index;done=false>>]
   [junit4]   2> 38369 T11 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/collection1/data/index
   [junit4]   2> 38381 T11 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
   [junit4]   2> 43383 T11 oas.SolrTestCaseJ4.tearDown ###Ending testReliability
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLBHttpSolrServer -Dtests.method=testReliability -Dtests.seed=2B5CEEF309D6B573 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=ja -Dtests.timezone=Europe/Brussels -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   37.9s J0 | TestLBHttpSolrServer.testReliability <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([2B5CEEF309D6B573:EA9433B5A8B064DA]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:539)
   [junit4]    > 	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
   [junit4]    > 	at org.apache.solr.client.solrj.TestLBHttpSolrServer.testReliability(TestLBHttpSolrServer.java:223)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:45317/solr
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:579)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:492)
   [junit4]    > 	... 43 more
   [junit4]    > Caused by: java.net.SocketTimeoutException: Read timed out
   [junit4]    > 	at java.net.SocketInputStream.socketRead0(Native Method)
   [junit4]    > 	at java.net.SocketInputStream.read(SocketInputStream.java:152)
   [junit4]    > 	at java.net.SocketInputStream.read(SocketInputStream.java:122)
   [junit4]    > 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
   [junit4]    > 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
   [junit4]    > 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
   [junit4]    > 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
   [junit4]    > 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
   [junit4]    > 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
   [junit4]    > 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
   [junit4]    > 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
   [junit4]    > 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:465)
   [junit4]    > 	... 46 more
   [junit4]   2> 43635 T11 oas.SolrTestCaseJ4.setUp ###Starting testTwoServers
   [junit4]   2> 43647 T11 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 43650 T11 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:48944
   [junit4]   2> 43652 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 43653 T11 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 43653 T11 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001
   [junit4]   2> 43654 T11 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/'
   [junit4]   2> 43705 T11 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/solr.xml
   [junit4]   2> 43819 T11 oasc.CoreContainer.<init> New CoreContainer 280752162
   [junit4]   2> 43820 T11 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/]
   [junit4]   2> 43822 T11 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 43823 T11 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 43824 T11 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 43825 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 43825 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 43826 T11 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 43827 T11 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 43827 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 43828 T11 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 43829 T11 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 43830 T11 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 43831 T11 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 43832 T11 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 43832 T11 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 43841 T60 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/'
   [junit4]   2> 43910 T60 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 43982 T60 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 43983 T60 oass.IndexSchema.readSchema Reading Solr Schema from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/conf/schema.xml
   [junit4]   2> 43997 T60 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 44028 T60 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 44030 T60 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/
   [junit4]   2> 44031 T60 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockFSDirectoryFactory
   [junit4]   2> 44032 T60 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/data/
   [junit4]   2> 44032 T60 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 44035 T60 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/data
   [junit4]   2> 44037 T60 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/data/index/
   [junit4]   2> 44038 T60 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 44040 T60 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/data/index
   [junit4]   2> 44051 T60 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@17aae3f3),segFN=segments_1,generation=1}
   [junit4]   2> 44052 T60 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 44055 T60 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 44056 T60 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 44057 T60 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 44058 T60 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 44058 T60 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 44059 T60 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 44060 T60 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 44060 T60 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 44061 T60 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 44062 T60 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 44063 T60 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 44086 T60 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 44089 T60 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 44092 T60 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 44096 T60 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 44100 T60 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 44101 T60 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 44105 T60 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@17aae3f3),segFN=segments_1,generation=1}
   [junit4]   2> 44106 T60 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 44107 T60 oass.SolrIndexSearcher.<init> Opening Searcher@37c8b85a[collection1] main
   [junit4]   2> 44108 T60 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/conf
   [junit4]   2> 44109 T60 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/conf}
   [junit4]   2> 44110 T60 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/conf
   [junit4]   2> 44111 T60 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 44112 T60 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using file:dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/conf
   [junit4]   2> 44113 T60 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 44114 T60 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 44114 T61 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@37c8b85a[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 44115 T60 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 44116 T11 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0
   [junit4]   2> 44117 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> ASYNC  NEW_CORE C5 name=collection1 org.apache.solr.core.SolrCore@20053644
   [junit4]   2> 44130 T53 C5 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]} 0 8
   [junit4]   2> 44136 T53 C5 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 44164 T53 C5 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@17aae3f3),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@17aae3f3),segFN=segments_2,generation=2}
   [junit4]   2> 44165 T53 C5 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 44172 T53 C5 oass.SolrIndexSearcher.<init> Opening Searcher@74bc6d07[collection1] main
   [junit4]  

[...truncated too long message...]

 Closing MockFSDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 77008 T11 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/data/index [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/data/index;done=false>>]
   [junit4]   2> 77009 T11 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/data/index
   [junit4]   2> 77029 T11 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/data [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/data;done=false>>]
   [junit4]   2> 77030 T11 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-0-001/collection1/data
   [junit4]   2> 77032 T11 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
   [junit4]   2> 82033 T11 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=920393858
   [junit4]   2> 82034 T11 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@651d3734
   [junit4]   2> 82035 T11 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 82036 T11 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 82036 T11 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 82037 T11 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 82041 T11 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 82042 T11 oasc.CachingDirectoryFactory.close Closing MockFSDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 82043 T11 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/collection1/data [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/collection1/data;done=false>>]
   [junit4]   2> 82044 T11 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/collection1/data
   [junit4]   2> 82045 T11 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/collection1/data/index [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/collection1/data/index;done=false>>]
   [junit4]   2> 82045 T11 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-1-001/collection1/data/index
   [junit4]   2> 82061 T11 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
   [junit4]   2> 87047 T11 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1744796127
   [junit4]   2> 87048 T11 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@3bd3ac38
   [junit4]   2> 87049 T11 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=10,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 87050 T11 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 87050 T11 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 87051 T11 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 87055 T11 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 87057 T11 oasc.CachingDirectoryFactory.close Closing MockFSDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 87058 T11 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/collection1/data/index [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/collection1/data/index;done=false>>]
   [junit4]   2> 87058 T11 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/collection1/data/index
   [junit4]   2> 87073 T11 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/collection1/data [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/collection1/data;done=false>>]
   [junit4]   2> 87074 T11 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001/instance-2-001/collection1/data
   [junit4]   2> 87076 T11 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
   [junit4]   2> 92068 T11 oas.SolrTestCaseJ4.tearDown ###Ending testSimple
   [junit4]   2> 92075 T11 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.TestLBHttpSolrServer-2B5CEEF309D6B573-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {name=PostingsFormat(name=LuceneFixedGap), id=PostingsFormat(name=LuceneFixedGap)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=ja, timezone=Europe/Brussels
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=3,free=66376992,total=226492416
   [junit4]   2> NOTE: All tests run in this JVM: [TestLBHttpSolrServer]
   [junit4] Completed on J0 in 93.23s, 3 tests, 1 error <<< FAILURES!

[...truncated 102 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/build.xml:525: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/build.xml:473: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build.xml:202: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/common-build.xml:508: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/lucene/common-build.xml:1363: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/lucene/common-build.xml:970: There were test failures: 54 suites, 330 tests, 1 error

Total time: 74 minutes 8 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-Tests-5.x-Java7 #2226
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 9312401 bytes
Compression is 0.0%
Took 3 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-5.x-Java7 - Build # 2229 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-5.x-Java7/2229/

All tests passed

Build Log:
[...truncated 32801 lines...]
-check-forbidden-all:
[forbidden-apis] Reading bundled API signatures: jdk-unsafe-1.7
[forbidden-apis] Reading bundled API signatures: jdk-deprecated-1.7
[forbidden-apis] Reading bundled API signatures: commons-io-unsafe-2.4
[forbidden-apis] Reading API signatures: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/lucene/tools/forbiddenApis/base.txt
[forbidden-apis] Reading API signatures: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/lucene/tools/forbiddenApis/servlet-api.txt
[forbidden-apis] Loading classes to check...
[forbidden-apis] Scanning for API signatures and dependencies...
[forbidden-apis] Forbidden method invocation: java.io.InputStreamReader#<init>(java.io.InputStream) [Uses default charset]
[forbidden-apis]   in org.apache.solr.core.SolrConfig (SolrConfig.java:356)
[forbidden-apis] Scanned 2289 (and 1562 related) class file(s) for forbidden API invocations (in 2.01s), 1 error(s).

BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/build.xml:525: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/build.xml:86: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build.xml:271: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/common-build.xml:491: Check for forbidden API calls failed, see log.

Total time: 82 minutes 40 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-Tests-5.x-Java7 #2226
Archived 1 artifacts
Archive block size is 32768
Received 0 blocks and 464 bytes
Compression is 0.0%
Took 36 ms
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure