You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "Eric Newton (JIRA)" <ji...@apache.org> on 2012/11/15 22:31:12 UTC
[jira] [Resolved] (ACCUMULO-834) thrift server responds slowly to
large numbers of requests
[ https://issues.apache.org/jira/browse/ACCUMULO-834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Newton resolved ACCUMULO-834.
----------------------------------
Resolution: Fixed
fixed with r1410025
> thrift server responds slowly to large numbers of requests
> ----------------------------------------------------------
>
> Key: ACCUMULO-834
> URL: https://issues.apache.org/jira/browse/ACCUMULO-834
> Project: Accumulo
> Issue Type: Bug
> Components: tserver
> Affects Versions: 1.4.1
> Environment: large scale cluster
> Reporter: Eric Newton
> Assignee: Keith Turner
> Priority: Minor
> Fix For: 1.5.0
>
>
> Testing on a large cluster, I found that the time to gather data from tablet servers got very long, resulting in "blocky" graphs in the monitor. After instrumenting the master, I found that each request to a tablet server was taking longer, and there were many servers. In the tserver logs, I saw lots of adjustments to the number of threads in the tablet server. At a large scale, say, hundreds of servers, and hundreds of test ingesters, sometimes there will be hundreds of threads requesting service. The thread pool grows slowly (one new thread per second when falling behind). Requests from the master starve waiting for new threads to be created.
> Possible fixes:
> * use a thread pool in the master to make status requests
> * use asynchronous messaging (the old ping/pong) to make status requests
> * modify the configuration to respond to large numbers of requests more quickly
> * increase the minimum number of threads available to service requests
> * replace the existing network infrastructure to use netty
> * modify the thread-pool checker to increase the number of threads more quickly when heavily loaded
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira