[jira] [Commented] (HDFS-9684) DataNode stopped sending heartbeat after getting OutOfMemoryError form DataTransfer thread.

["Walter Su (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-9684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15194914#comment-15194914 ] 

Walter Su commented on HDFS-9684:
---------------------------------

I have seen a case DN got command from NN to transfer huge numbers of blocks. There's 7000+ threads at its peak. I don't  advocate recover from {{OutOfMemoryError}}. But it's our responsibility not to create too much threads at first place.

> DataNode stopped sending heartbeat after getting OutOfMemoryError form DataTransfer thread.
> -------------------------------------------------------------------------------------------
>
>                 Key: HDFS-9684
>                 URL: https://issues.apache.org/jira/browse/HDFS-9684
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>    Affects Versions: 2.7.1
>            Reporter: Surendra Singh Lilhore
>            Assignee: Surendra Singh Lilhore
>            Priority: Blocker
>         Attachments: HDFS-9684.01.patch
>
>
> {noformat}
> java.lang.OutOfMemoryError: unable to create new native thread
> 	at java.lang.Thread.start0(Native Method)
> 	at java.lang.Thread.start(Thread.java:714)
> 	at org.apache.hadoop.hdfs.server.datanode.DataNode.transferBlock(DataNode.java:1999)
> 	at org.apache.hadoop.hdfs.server.datanode.DataNode.transferBlocks(DataNode.java:2008)
> 	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:657)
> 	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:615)
> 	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:857)
> 	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:671)
> 	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:823)
> 	at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)