You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by "Kihwal Lee (JIRA)" <ji...@apache.org> on 2016/02/26 16:15:18 UTC
[jira] [Resolved] (HDFS-9863) DataNode doesn't log any shutdown
info when the process of DataNode exiting
[ https://issues.apache.org/jira/browse/HDFS-9863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kihwal Lee resolved HDFS-9863.
------------------------------
Resolution: Invalid
> DataNode doesn't log any shutdown info when the process of DataNode exiting
> ---------------------------------------------------------------------------
>
> Key: HDFS-9863
> URL: https://issues.apache.org/jira/browse/HDFS-9863
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.7.1
> Reporter: Lin Yiqun
> Attachments: datanode-restart_after.gc.log, datanode-restart_before.gc.log, datanode.log
>
>
> One of my datanodes exited without any shutdown info.
> {code}
> 2016-02-25 14:46:00,283 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-1942012336-XX.XX.2.191-1406726500544:blk_1730224536_658031130, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
> 2016-02-25 15:03:55,639 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG: host = XX.XX6032/XX.XX.6.32
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 2.7.1
> {code}
> I think maybe full gc causes this problem, so I looked the datanode gc log. There is a cms gc but the time of this gc is after than restart datanode time.
> {code}
> 2016-02-25T15:03:57.930+0800: 2.756: [GC2016-02-25T15:03:57.930+0800: 2.756: [ParNew: 1677824K->24417K(1887488K), 0.0249280 secs] 1677824K->24417K(8178944K), 0.0251010 secs] [Times: user=0.24 sys=0.07, real=0.02 secs]
> 2016-02-25T15:12:46.498+0800: 531.324: [GC [1 CMS-initial-mark: 0K(6291456K)] 780481K(8178944K), 0.0554170 secs] [Times: user=0.06 sys=0.00, real=0.07 secs]
> 2016-02-25T15:12:46.567+0800: 531.393: [CMS-concurrent-mark-start]
> 2016-02-25T15:12:46.574+0800: 531.400: [CMS-concurrent-mark: 0.006/0.007 secs] [Times: user=0.07 sys=0.02, real=0.01 secs]
> 2016-02-25T15:12:46.574+0800: 531.400: [CMS-concurrent-preclean-start]
> 2016-02-25T15:12:46.589+0800: 531.415: [CMS-concurrent-preclean: 0.015/0.015 secs] [Times: user=0.16 sys=0.06, real=0.01 secs]
> {code}
> It seems this is not the main reason. Gc of time before datanode exiting seems normal.
> {code}
> 2016-02-25T14:45:39.743+0800: 5431411.796: [GC2016-02-25T14:45:39.743+0800: 5431411.796: [ParNew: 1686799K->22696K(1887488K), 0.0385700 secs] 2908579K->1244476K(8178944K) icms_dc=0 , 0.0388280 secs] [Times: user=0.23 sys=0.01, real=0.04 secs]
> {code}
> So it looks confusion. Attach the complete gc logs and datanode log.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)