You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Christian Kunz (JIRA)" <ji...@apache.org> on 2008/11/26 23:36:46 UTC
[jira] Created: (HADOOP-4735) NameNode reporting 0 size for
originally non-empty files
NameNode reporting 0 size for originally non-empty files
--------------------------------------------------------
Key: HADOOP-4735
URL: https://issues.apache.org/jira/browse/HADOOP-4735
Project: Hadoop Core
Issue Type: Bug
Components: dfs
Affects Versions: 0.17.2
Reporter: Christian Kunz
NameNode reports 0 size for a handful of files that were non-empty originally.
The corresponding blocks on the DataNodes are non-empty.
NameNode must have reported correct size at some time, because applications that would have failed with 0 size files, executed successfully.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4735) NameNode reporting 0 size for
originally non-empty files
Posted by "Christian Kunz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12651186#action_12651186 ]
Christian Kunz commented on HADOOP-4735:
----------------------------------------
They were created 11/16 (I found the corresponding log messages on the Namenode, no other log messages referring to these blocks) and used as non-empty files by subsequent jobs (non-empty otherwise they would have failed as mentioned in the description). They must have been become empty within the last few days.
I checked one of the blocks on one of the datanodes after namenode restart. Still there and still non-empty.
I will now move the empty files away into a different location (they will still be available for examination) and re-create the original files from the non-empty blocks.
> NameNode reporting 0 size for originally non-empty files
> --------------------------------------------------------
>
> Key: HADOOP-4735
> URL: https://issues.apache.org/jira/browse/HADOOP-4735
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.2
> Reporter: Christian Kunz
>
> NameNode reports 0 size for a handful of files that were non-empty originally.
> The corresponding blocks on the DataNodes are non-empty.
> NameNode must have reported correct size at some time, because applications that would have failed with 0 size files, executed successfully.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-4735) NameNode reporting 0 size for
originally non-empty files
Posted by "Christian Kunz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christian Kunz resolved HADOOP-4735.
------------------------------------
Resolution: Cannot Reproduce
Currently no longer seen.
> NameNode reporting 0 size for originally non-empty files
> --------------------------------------------------------
>
> Key: HADOOP-4735
> URL: https://issues.apache.org/jira/browse/HADOOP-4735
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.2
> Reporter: Christian Kunz
>
> NameNode reports 0 size for a handful of files that were non-empty originally.
> The corresponding blocks on the DataNodes are non-empty.
> NameNode must have reported correct size at some time, because applications that would have failed with 0 size files, executed successfully.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4735) NameNode reporting 0 size for
originally non-empty files
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12651179#action_12651179 ]
Konstantin Shvachko commented on HADOOP-4735:
---------------------------------------------
> I restarted the namenode, with no success.
Do you still see the blocks on the data-nodes after restarting?
In 0.17 there was no so called lease recovery for files that were not closed properly.
Do you know whether the 0-size files were closed before becoming empty?
> NameNode reporting 0 size for originally non-empty files
> --------------------------------------------------------
>
> Key: HADOOP-4735
> URL: https://issues.apache.org/jira/browse/HADOOP-4735
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.2
> Reporter: Christian Kunz
>
> NameNode reports 0 size for a handful of files that were non-empty originally.
> The corresponding blocks on the DataNodes are non-empty.
> NameNode must have reported correct size at some time, because applications that would have failed with 0 size files, executed successfully.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4735) NameNode reporting 0 size for
originally non-empty files
Posted by "Christian Kunz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12651404#action_12651404 ]
Christian Kunz commented on HADOOP-4735:
----------------------------------------
All 0 size files had similar log messages like (initial size 0 --> update to non-zero during closing)
2008-11-16 22:52:04,302 INFO org.apache.hadoop.dfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: xxx.yyy.zzz.vvv:port is added to blk_6021851155145233808 size 0
2008-11-16 22:52:04,540 WARN org.apache.hadoop.dfs.StateChange: BLOCK* NameSystem.addStoredBlock: Redundant addStoredBlock request received for blk_6021851155145233808 on xxx.yyy.zzz.vvv:port size 38807484
Also I checked that the file corruption started after a single namenode restart after the file creation.
So I would conclude that the edits file must have contained the initial 0 size instead of the final size. The fact that the files continue to exist after the namenode restart indicates that they got properly closed. Correct? Unfortunately, there are no log messages about closing the file.
I start to suspect that there is rare race condition between updating the block with the correct size and storing it to the edits file during the closing process. Would that be possible?
> NameNode reporting 0 size for originally non-empty files
> --------------------------------------------------------
>
> Key: HADOOP-4735
> URL: https://issues.apache.org/jira/browse/HADOOP-4735
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.2
> Reporter: Christian Kunz
>
> NameNode reports 0 size for a handful of files that were non-empty originally.
> The corresponding blocks on the DataNodes are non-empty.
> NameNode must have reported correct size at some time, because applications that would have failed with 0 size files, executed successfully.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.