[jira] Commented: (HADOOP-2910) Throttle IPC Client/Server during bursts of requests or server slowdown

["Hairong Kuang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-2910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12576448#action_12576448 ] 

Hairong Kuang commented on HADOOP-2910:
---------------------------------------

> Limit call queue length or limit the amount of memory used in the call queue. 

Currently each IPC request has a header indicating the length of the packet. IPC server reads the length, then allocates a buffer with a size of the length, reads the request into the buffer, marshals it, then put the marshaled request into the call queue. If we want to limit the amount of memory used in the call queue, we can keep a counter tracking the total size of the unmarshaled requests. Whenever the server reads a request from the socket into the buffer, increment the counter. When a handler takes the request out of the queue, decrement the counter. It stops reading from the socket if the incoming request will make the counter to exceed the max size limit.


> Throttle IPC Client/Server during bursts of requests or server slowdown
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-2910
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2910
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.16.0
>            Reporter: Hairong Kuang
>             Fix For: 0.17.0
>
>
> I propose the following to avoid an IPC server being swarmed by too many requests and connections
> 1. Limit call queue length or limit the amount of memory used in the call queue. This can be done by including the size of a request in the header and storing unmarshaled requests in the call queue. 
> 2. If the call queue is full or queue buffer is full, stop reading requests from sockets. So requests stay at the server's system buffer or at the client side and thus eventually throttle the client. 
> 3. Limit the total number of connections. Do not accept new connections if the connection limit is exceeded. (Note: this solution is unfair to new connections.) 
> 4. If receive out of memory exception, close the current connection. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.