You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Jim Huang (JIRA)" <ji...@apache.org> on 2008/07/14 23:36:31 UTC
[jira] Created: (HADOOP-3758) Excessive exceptions in HDFS namenode
log file
Excessive exceptions in HDFS namenode log file
----------------------------------------------
Key: HADOOP-3758
URL: https://issues.apache.org/jira/browse/HADOOP-3758
Project: Hadoop Core
Issue Type: Bug
Components: dfs
Affects Versions: 0.17.1
Reporter: Jim Huang
I upgraded a big cluster, out of which 10 nodes did not get upgraded.
The namenode log showed excessive exceptions, causing the namenode log to ate the entire partition space, in this case close to 700GB log file was generated on the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-3758) Excessive exceptions in
HDFS namenode log file
Posted by "Lohit Vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12613510#action_12613510 ]
lohit edited comment on HADOOP-3758 at 7/14/08 11:12 PM:
--------------------------------------------------------------------
Patch for trunk. I tested this by changing layout version and trying to start a datanode connecting to namenode. It fails with IncorrectVersionException.
was (Author: lohit):
Patch for trunk. I tested this my changing layout version and trying to connect a datanode. It fails with IncorrectVersionException.
> Excessive exceptions in HDFS namenode log file
> ----------------------------------------------
>
> Key: HADOOP-3758
> URL: https://issues.apache.org/jira/browse/HADOOP-3758
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.1
> Reporter: Jim Huang
> Assignee: Lohit Vijayarenu
> Priority: Blocker
> Attachments: HADOOP-3758-18.patch, HADOOP-3758-trunk.patch
>
>
> I upgraded a big cluster, out of which 10 nodes did not get upgraded.
> The namenode log showed excessive exceptions, causing the namenode log to ate the entire partition space, in this case close to 700GB log file was generated on the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3758) Excessive exceptions in HDFS namenode
log file
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3758:
----------------------------------
Resolution: Fixed
Fix Version/s: 0.17.2
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Lohit
> Excessive exceptions in HDFS namenode log file
> ----------------------------------------------
>
> Key: HADOOP-3758
> URL: https://issues.apache.org/jira/browse/HADOOP-3758
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.1
> Reporter: Jim Huang
> Assignee: Lohit Vijayarenu
> Priority: Blocker
> Fix For: 0.17.2
>
> Attachments: HADOOP-3758-17.patch, HADOOP-3758-18.patch, HADOOP-3758-trunk.patch, HADOOP-3758-trunk.patch
>
>
> I upgraded a big cluster, out of which 10 nodes did not get upgraded.
> The namenode log showed excessive exceptions, causing the namenode log to ate the entire partition space, in this case close to 700GB log file was generated on the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3758) Excessive exceptions in HDFS
namenode log file
Posted by "Lohit Vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12613543#action_12613543 ]
Lohit Vijayarenu commented on HADOOP-3758:
------------------------------------------
While patch is waiting on hudson, I ran the tests on my LINUX box. All tests pass, even test-patch. There is not testcase, did manual testing.
> Excessive exceptions in HDFS namenode log file
> ----------------------------------------------
>
> Key: HADOOP-3758
> URL: https://issues.apache.org/jira/browse/HADOOP-3758
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.1
> Reporter: Jim Huang
> Assignee: Lohit Vijayarenu
> Priority: Blocker
> Attachments: HADOOP-3758-18.patch, HADOOP-3758-trunk.patch
>
>
> I upgraded a big cluster, out of which 10 nodes did not get upgraded.
> The namenode log showed excessive exceptions, causing the namenode log to ate the entire partition space, in this case close to 700GB log file was generated on the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3758) Excessive exceptions in HDFS namenode
log file
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-3758:
------------------------------------
Priority: Blocker (was: Major)
Assignee: Lohit Vijayarenu
> Excessive exceptions in HDFS namenode log file
> ----------------------------------------------
>
> Key: HADOOP-3758
> URL: https://issues.apache.org/jira/browse/HADOOP-3758
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.1
> Reporter: Jim Huang
> Assignee: Lohit Vijayarenu
> Priority: Blocker
>
> I upgraded a big cluster, out of which 10 nodes did not get upgraded.
> The namenode log showed excessive exceptions, causing the namenode log to ate the entire partition space, in this case close to 700GB log file was generated on the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3758) Excessive exceptions in HDFS namenode
log file
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3758:
---------------------------------
Hadoop Flags: [Reviewed]
+1. Looks good.
> Excessive exceptions in HDFS namenode log file
> ----------------------------------------------
>
> Key: HADOOP-3758
> URL: https://issues.apache.org/jira/browse/HADOOP-3758
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.1
> Reporter: Jim Huang
> Assignee: Lohit Vijayarenu
> Priority: Blocker
> Attachments: HADOOP-3758-17.patch, HADOOP-3758-18.patch, HADOOP-3758-trunk.patch, HADOOP-3758-trunk.patch
>
>
> I upgraded a big cluster, out of which 10 nodes did not get upgraded.
> The namenode log showed excessive exceptions, causing the namenode log to ate the entire partition space, in this case close to 700GB log file was generated on the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3758) Excessive exceptions in HDFS
namenode log file
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12613472#action_12613472 ]
Raghu Angadi commented on HADOOP-3758:
--------------------------------------
Thats pretty painful. We should include "IncorrectVersionException" as one of the fatal exceptions at the datanode.
See {{DataNode.java:offserService()}} :
{noformat}
} catch(RemoteException re) {
String reClass = re.getClassName();
if (UnregisteredDatanodeException.class.getName().equals(reClass) ||
DisallowedDatanodeException.class.getName().equals(reClass)) {
LOG.warn("DataNode is shutting down: " +
StringUtils.stringifyException(re));
shutdown();
return;
}
{noformat}
> Excessive exceptions in HDFS namenode log file
> ----------------------------------------------
>
> Key: HADOOP-3758
> URL: https://issues.apache.org/jira/browse/HADOOP-3758
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.1
> Reporter: Jim Huang
>
> I upgraded a big cluster, out of which 10 nodes did not get upgraded.
> The namenode log showed excessive exceptions, causing the namenode log to ate the entire partition space, in this case close to 700GB log file was generated on the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-3758) Excessive exceptions in
HDFS namenode log file
Posted by "Jim Huang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12613460#action_12613460 ]
jimhuang edited comment on HADOOP-3758 at 7/14/08 3:01 PM:
------------------------------------------------------------
In the span of 11 seconds, there were 103,367 exceptions generated for only 10 unique nodes that had incorrect versions.
Here the repetitive log entries that ate up all the disk space.
{noformat}
2008-06-26 22:48:18,952 INFO org.apache.hadoop.ipc.Server: IPC Server handler 19 on 8020, call sendHeartbeat(A.B.C.D:50010, 2971509878784, 1731424256, 2150426691690, 0, 0) from A.B.C.D:43226: error: org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data node. Reported: -11. Expecting = -13.
org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data node. Reported: -11. Expecting = -13.
at org.apache.hadoop.dfs.NameNode.verifyVersion(NameNode.java:682)
at org.apache.hadoop.dfs.NameNode.verifyRequest(NameNode.java:669)
at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:557)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
2008-06-26 22:48:18,953 INFO org.apache.hadoop.ipc.Server: IPC Server handler 27 on 8020, call sendHeartbeat(A.B.C.E:50010, 2971509878784, 1993637888, 2151120783483, 0, 0) from A.B.C.E:56503: error: org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data node. Reported: -11. Expecting = -13.
org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data node. Reported: -11. Expecting = -13.
at org.apache.hadoop.dfs.NameNode.verifyVersion(NameNode.java:682)
at org.apache.hadoop.dfs.NameNode.verifyRequest(NameNode.java:669)
at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:557)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
{noformat}
was (Author: jimhuang):
In the span of 11 seconds, there were 103,367 exceptions generated for only 10 unique nodes that had incorrect versions.
Here the repetitive log entries that ate up all the disk space.
{noformat}
2008-06-26 22:48:18,952 INFO org.apache.hadoop.ipc.Server: IPC Server handler 19 on 8020, call sendHeartbeat(A.B.C.D:50010, 2971509878784, 1731424256, 2150426691690, 0, 0) from 76.1
3.169.226:43226: error: org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data node. Reported: -11. Expecting = -13.
org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data node. Reported: -11. Expecting = -13.
at org.apache.hadoop.dfs.NameNode.verifyVersion(NameNode.java:682)
at org.apache.hadoop.dfs.NameNode.verifyRequest(NameNode.java:669)
at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:557)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
2008-06-26 22:48:18,953 INFO org.apache.hadoop.ipc.Server: IPC Server handler 27 on 8020, call sendHeartbeat(A.B.C.E:50010, 2971509878784, 1993637888, 2151120783483, 0, 0) from 76.1
3.169.218:56503: error: org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data node. Reported: -11. Expecting = -13.
org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data node. Reported: -11. Expecting = -13.
at org.apache.hadoop.dfs.NameNode.verifyVersion(NameNode.java:682)
at org.apache.hadoop.dfs.NameNode.verifyRequest(NameNode.java:669)
at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:557)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
{noformat}
> Excessive exceptions in HDFS namenode log file
> ----------------------------------------------
>
> Key: HADOOP-3758
> URL: https://issues.apache.org/jira/browse/HADOOP-3758
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.1
> Reporter: Jim Huang
>
> I upgraded a big cluster, out of which 10 nodes did not get upgraded.
> The namenode log showed excessive exceptions, causing the namenode log to ate the entire partition space, in this case close to 700GB log file was generated on the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3758) Excessive exceptions in HDFS namenode
log file
Posted by "Lohit Vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lohit Vijayarenu updated HADOOP-3758:
-------------------------------------
Attachment: HADOOP-3758-trunk.patch
Patch for trunk. I tested this my changing layout version and trying to connect a datanode. It fails with IncorrectVersionException.
> Excessive exceptions in HDFS namenode log file
> ----------------------------------------------
>
> Key: HADOOP-3758
> URL: https://issues.apache.org/jira/browse/HADOOP-3758
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.1
> Reporter: Jim Huang
> Assignee: Lohit Vijayarenu
> Priority: Blocker
> Attachments: HADOOP-3758-18.patch, HADOOP-3758-trunk.patch
>
>
> I upgraded a big cluster, out of which 10 nodes did not get upgraded.
> The namenode log showed excessive exceptions, causing the namenode log to ate the entire partition space, in this case close to 700GB log file was generated on the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3758) Excessive exceptions in HDFS
namenode log file
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12624706#action_12624706 ]
Hudson commented on HADOOP-3758:
--------------------------------
Integrated in Hadoop-trunk #581 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/581/])
> Excessive exceptions in HDFS namenode log file
> ----------------------------------------------
>
> Key: HADOOP-3758
> URL: https://issues.apache.org/jira/browse/HADOOP-3758
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.1
> Reporter: Jim Huang
> Assignee: Lohit Vijayarenu
> Priority: Blocker
> Fix For: 0.17.2
>
> Attachments: HADOOP-3758-17.patch, HADOOP-3758-18.patch, HADOOP-3758-trunk.patch, HADOOP-3758-trunk.patch
>
>
> I upgraded a big cluster, out of which 10 nodes did not get upgraded.
> The namenode log showed excessive exceptions, causing the namenode log to ate the entire partition space, in this case close to 700GB log file was generated on the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3758) Excessive exceptions in HDFS
namenode log file
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12613496#action_12613496 ]
Raghu Angadi commented on HADOOP-3758:
--------------------------------------
In addition, DN should update lastHeartBeat time even if sendHeartbeat() results in an exception.. this will avoid similar problems with future errors.
> Excessive exceptions in HDFS namenode log file
> ----------------------------------------------
>
> Key: HADOOP-3758
> URL: https://issues.apache.org/jira/browse/HADOOP-3758
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.1
> Reporter: Jim Huang
> Assignee: Lohit Vijayarenu
> Priority: Blocker
>
> I upgraded a big cluster, out of which 10 nodes did not get upgraded.
> The namenode log showed excessive exceptions, causing the namenode log to ate the entire partition space, in this case close to 700GB log file was generated on the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3758) Excessive exceptions in HDFS namenode
log file
Posted by "Lohit Vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lohit Vijayarenu updated HADOOP-3758:
-------------------------------------
Attachment: HADOOP-3758-18.patch
Patch for 0.18
> Excessive exceptions in HDFS namenode log file
> ----------------------------------------------
>
> Key: HADOOP-3758
> URL: https://issues.apache.org/jira/browse/HADOOP-3758
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.1
> Reporter: Jim Huang
> Assignee: Lohit Vijayarenu
> Priority: Blocker
> Attachments: HADOOP-3758-18.patch, HADOOP-3758-trunk.patch
>
>
> I upgraded a big cluster, out of which 10 nodes did not get upgraded.
> The namenode log showed excessive exceptions, causing the namenode log to ate the entire partition space, in this case close to 700GB log file was generated on the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3758) Excessive exceptions in HDFS namenode
log file
Posted by "Lohit Vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lohit Vijayarenu updated HADOOP-3758:
-------------------------------------
Attachment: HADOOP-3758-trunk.patch
> Excessive exceptions in HDFS namenode log file
> ----------------------------------------------
>
> Key: HADOOP-3758
> URL: https://issues.apache.org/jira/browse/HADOOP-3758
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.1
> Reporter: Jim Huang
> Assignee: Lohit Vijayarenu
> Priority: Blocker
> Attachments: HADOOP-3758-17.patch, HADOOP-3758-18.patch, HADOOP-3758-trunk.patch, HADOOP-3758-trunk.patch
>
>
> I upgraded a big cluster, out of which 10 nodes did not get upgraded.
> The namenode log showed excessive exceptions, causing the namenode log to ate the entire partition space, in this case close to 700GB log file was generated on the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3758) Excessive exceptions in HDFS namenode
log file
Posted by "Lohit Vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lohit Vijayarenu updated HADOOP-3758:
-------------------------------------
Attachment: HADOOP-3758-17.patch
patch for 0.17.2
> Excessive exceptions in HDFS namenode log file
> ----------------------------------------------
>
> Key: HADOOP-3758
> URL: https://issues.apache.org/jira/browse/HADOOP-3758
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.1
> Reporter: Jim Huang
> Assignee: Lohit Vijayarenu
> Priority: Blocker
> Attachments: HADOOP-3758-17.patch, HADOOP-3758-18.patch, HADOOP-3758-trunk.patch, HADOOP-3758-trunk.patch
>
>
> I upgraded a big cluster, out of which 10 nodes did not get upgraded.
> The namenode log showed excessive exceptions, causing the namenode log to ate the entire partition space, in this case close to 700GB log file was generated on the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3758) Excessive exceptions in HDFS namenode
log file
Posted by "Lohit Vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lohit Vijayarenu updated HADOOP-3758:
-------------------------------------
Status: Patch Available (was: Open)
> Excessive exceptions in HDFS namenode log file
> ----------------------------------------------
>
> Key: HADOOP-3758
> URL: https://issues.apache.org/jira/browse/HADOOP-3758
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.1
> Reporter: Jim Huang
> Assignee: Lohit Vijayarenu
> Priority: Blocker
> Attachments: HADOOP-3758-18.patch, HADOOP-3758-trunk.patch
>
>
> I upgraded a big cluster, out of which 10 nodes did not get upgraded.
> The namenode log showed excessive exceptions, causing the namenode log to ate the entire partition space, in this case close to 700GB log file was generated on the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3758) Excessive exceptions in HDFS
namenode log file
Posted by "Jim Huang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12613460#action_12613460 ]
Jim Huang commented on HADOOP-3758:
-----------------------------------
In the span of 11 seconds, there were 103,367 exceptions generated for only 10 unique nodes that had incorrect versions.
Here the repetitive log entries that ate up all the disk space.
{noformat}
2008-06-26 22:48:18,952 INFO org.apache.hadoop.ipc.Server: IPC Server handler 19 on 8020, call sendHeartbeat(A.B.C.D:50010, 2971509878784, 1731424256, 2150426691690, 0, 0) from 76.1
3.169.226:43226: error: org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data node. Reported: -11. Expecting = -13.
org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data node. Reported: -11. Expecting = -13.
at org.apache.hadoop.dfs.NameNode.verifyVersion(NameNode.java:682)
at org.apache.hadoop.dfs.NameNode.verifyRequest(NameNode.java:669)
at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:557)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
2008-06-26 22:48:18,953 INFO org.apache.hadoop.ipc.Server: IPC Server handler 27 on 8020, call sendHeartbeat(A.B.C.E:50010, 2971509878784, 1993637888, 2151120783483, 0, 0) from 76.1
3.169.218:56503: error: org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data node. Reported: -11. Expecting = -13.
org.apache.hadoop.dfs.IncorrectVersionException: Unexpected version of data node. Reported: -11. Expecting = -13.
at org.apache.hadoop.dfs.NameNode.verifyVersion(NameNode.java:682)
at org.apache.hadoop.dfs.NameNode.verifyRequest(NameNode.java:669)
at org.apache.hadoop.dfs.NameNode.sendHeartbeat(NameNode.java:557)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
{noformat}
> Excessive exceptions in HDFS namenode log file
> ----------------------------------------------
>
> Key: HADOOP-3758
> URL: https://issues.apache.org/jira/browse/HADOOP-3758
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.1
> Reporter: Jim Huang
>
> I upgraded a big cluster, out of which 10 nodes did not get upgraded.
> The namenode log showed excessive exceptions, causing the namenode log to ate the entire partition space, in this case close to 700GB log file was generated on the namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.