You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by "Jiandan Yang (JIRA)" <ji...@apache.org> on 2017/09/07 02:29:00 UTC
[jira] [Created] (YARN-7168) The size of dataQueue and ackQueue in
DataStreamer has no limit when writer thread is interrupted
Jiandan Yang created YARN-7168:
-----------------------------------
Summary: The size of dataQueue and ackQueue in DataStreamer has no limit when writer thread is interrupted
Key: YARN-7168
URL: https://issues.apache.org/jira/browse/YARN-7168
Project: Hadoop YARN
Issue Type: Bug
Components: client
Reporter: Jiandan Yang
In our cluster, when found NodeManager frequently FullGC when decommissioning NodeManager, and we found the biggest object is dataQueue of DataStreamer, it has almost 6w DFSPacket, and every DFSPacket is about 64k.
!mat.jpg|memory_analysis!
The root reason is that the size of dataQueue and ackQueue in DataStreamer has no limit when writer thread is interrupted. I know NodeManager may stop writing when interruped, but DFSOutputStream also could do something to avoid fullgc
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org