You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Daryn Sharp (JIRA)" <ji...@apache.org> on 2014/09/04 18:26:52 UTC

[jira] [Commented] (HDFS-6967) DNs may OOM under high webhdfs load

    [ https://issues.apache.org/jira/browse/HDFS-6967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14121518#comment-14121518 ] 

Daryn Sharp commented on HDFS-6967:
-----------------------------------

This is becoming a critical issue.  Wide and/or many jobs overwhelm DNs (due to smaller heap) and sometimes the NN.  Either can OOM or go into full GC.  In the case of DNs, it can GC so much that the NN declares it dead.

Hadoop's jetty version does not support limiting of connections, in fact the code contains "README"s that connections should be limited down in the bowels of private/anonymous classes.  It would require some serious risky hacks to override jetty's behavior.

I'd like to solicit comments on an internal approach we are trying:  rather than redirect clients to a DN with a replica, redirect to a random DN in a rack with a replica.  This distributes the jetty connections, DFSClients, and associated objects over many more nodes with an insignificant impact to throughput - esp. over higher latency links and compared to the time it takes for the client to timeout the connection and retry another node which may also be overloaded.

> DNs may OOM under high webhdfs load
> -----------------------------------
>
>                 Key: HDFS-6967
>                 URL: https://issues.apache.org/jira/browse/HDFS-6967
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode, webhdfs
>    Affects Versions: 2.0.0-alpha, 3.0.0
>            Reporter: Daryn Sharp
>            Assignee: Eric Payne
>
> Webhdfs uses jetty.  The size of the request thread pool is limited, but jetty will accept and queue infinite connections.  Every queued connection is "heavy" with buffers, etc.  Unlike data streamer connections, thousands of webhdfs connections will quickly OOM a DN.  The accepted requests must be bounded and excess clients rejected so they retry on a new DN.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)