[jira] Issue Comment Edited: (HADOOP-2910) Throttle IPC Client/Server during bursts of requests or server slowdown

["Raghu Angadi (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-2910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587763#action_12587763 ] 

rangadi edited comment on HADOOP-2910 at 4/10/08 12:48 PM:
----------------------------------------------------------------

If I could add one more to this jira, I would increase the queue size to at least 15k or so (or multiply handlers by 250-300).

Should I file another jira for handling file descriptor limits?

Regd windows:  we should add in FAQ discouraging users from using standard windows for large clusters (especially as NameNode or JobTracker etc.). We should recommend Windows server. We don't need to try hard to write a good server when the OS deliberately tries to avoid it.

bq. To me this indicates that the backlog is effective up to around 16k connections, but that connections queued this way will timeout after four minutes.

I don't think 16k matters at all.. even if we start 100k client or 2k clients, most of them will timeout around the same time, because beyond certain limit, kernel just ignores the SYN packets. This is what happens on my Linux box :

# It accepts 131 connections immediately.
# Then accepts 2 connections more every 9-10 seconds.
# Most clients will timeout around 3-4 minutes.




      was (Author: rangadi):
    
If I could add one more to this jira, I would increase the queue size to at least 15k or so (or multiply handlers by 250-300).

Should I file another jira for handling file descriptor limits?

Regd windows:  we should add in FAQ discouraging users from using standard windows for large clusters (especially as NameNode or JobTracker etc.). We should recommend Windows server. We don't need to try hard to write a good server when the OS deliberately tries to avoid it.

bq. To me this indicates that the backlog is effective up to around 16k connections, but that connections queued this way will timeout after four minutes.

I don't think 16k matters at all.. even if we start 100k client or 2k clients, most of them will timeout around the same time. This is what happens on my Linux box :

# It accepts 131 connections immediately.
# Then accepts 2 connections more every 9-10 seconds.
# Most clients will timeout around 3-4 minutes.



  
> Throttle IPC Client/Server during bursts of requests or server slowdown
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-2910
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2910
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.16.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.18.0
>
>         Attachments: callQueue.patch, callQueue1.patch, callQueue2.patch, callQueue3.patch, TestBacklog.java, TestBacklog.java, TestBacklogWithPool.java, throttleClient.patch
>
>
> I propose the following to avoid an IPC server being swarmed by too many requests and connections
> 1. Limit call queue length or limit the amount of memory used in the call queue. This can be done by including the size of a request in the header and storing unmarshaled requests in the call queue. 
> 2. If the call queue is full or queue buffer is full, stop reading requests from sockets. So requests stay at the server's system buffer or at the client side and thus eventually throttle the client. 
> 3. Limit the total number of connections. Do not accept new connections if the connection limit is exceeded. (Note: this solution is unfair to new connections.) 
> 4. If receive out of memory exception, close the current connection. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.