You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Christian Kunz (JIRA)" <ji...@apache.org> on 2007/07/13 22:28:04 UTC
[jira] Created: (HADOOP-1612) listing of an output directory
shortly after job completion fails
listing of an output directory shortly after job completion fails
-----------------------------------------------------------------
Key: HADOOP-1612
URL: https://issues.apache.org/jira/browse/HADOOP-1612
Project: Hadoop
Issue Type: Bug
Components: mapred
Affects Versions: 0.14.0
Reporter: Christian Kunz
Sometimes, after a job finishes, and another application wants to rename dfs files created by that job, listing of the output directory containing the newly created files fails. File creation and directory listing is done via libhdfs, but it is unlikely that this makes any difference, therefore, I add this to the mapred component.
It might be a race condition: does the job complete before the files in the output directory are promoted?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-1612) listing of an output directory
shortly after job completion fails
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley resolved HADOOP-1612.
-----------------------------------
Resolution: Won't Fix
This seems to have been caused by two parts:
1. HADOOP-1576
2. Non-winning speculative execution tasks are not cleaned up before the job is marked complete. This is difficult to fix, because you need to wait for the task to report in, which may take more than 10 minutes in the case of an unresponsive task tracker. Users should work around this by ensuring that they ignore all contents of the output directory that start with "_".
> listing of an output directory shortly after job completion fails
> -----------------------------------------------------------------
>
> Key: HADOOP-1612
> URL: https://issues.apache.org/jira/browse/HADOOP-1612
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.14.0
> Reporter: Christian Kunz
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.14.0
>
>
> Sometimes, after a job finishes, and another application wants to rename dfs files created by that job, listing of the output directory containing the newly created files fails. File creation and directory listing is done via libhdfs, but it is unlikely that this makes any difference, therefore, I add this to the mapred component.
> It might be a race condition: does the job complete before the files in the output directory are promoted?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (HADOOP-1612) listing of an output directory
shortly after job completion fails
Posted by "Christian Kunz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christian Kunz reopened HADOOP-1612:
------------------------------------
> listing of an output directory shortly after job completion fails
> -----------------------------------------------------------------
>
> Key: HADOOP-1612
> URL: https://issues.apache.org/jira/browse/HADOOP-1612
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.14.0
> Reporter: Christian Kunz
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.14.0
>
>
> Sometimes, after a job finishes, and another application wants to rename dfs files created by that job, listing of the output directory containing the newly created files fails. File creation and directory listing is done via libhdfs, but it is unlikely that this makes any difference, therefore, I add this to the mapred component.
> It might be a race condition: does the job complete before the files in the output directory are promoted?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1612) listing of an output directory
shortly after job completion fails
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-1612:
----------------------------------
Assignee: Arun C Murthy
bq. It might be a race condition: does the job complete before the files in the output directory are promoted?
No. The way it works is that a task isn't declared as a *success* until promotion completes successfully (to guard against failure to move files etc.) and hence the job is declared a *success* only after tasks promote their files successfully.
I'll take this one up... and I'd appreciate any more information you have to help me fix this. For e.g. do you have multiple outputs per task i.e. do you open a hdfs file from within your map/reduce functions and write to it etc.? Thanks!
> listing of an output directory shortly after job completion fails
> -----------------------------------------------------------------
>
> Key: HADOOP-1612
> URL: https://issues.apache.org/jira/browse/HADOOP-1612
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.14.0
> Reporter: Christian Kunz
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.14.0
>
>
> Sometimes, after a job finishes, and another application wants to rename dfs files created by that job, listing of the output directory containing the newly created files fails. File creation and directory listing is done via libhdfs, but it is unlikely that this makes any difference, therefore, I add this to the mapred component.
> It might be a race condition: does the job complete before the files in the output directory are promoted?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1612) listing of an output directory
shortly after job completion fails
Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sameer Paranjpye updated HADOOP-1612:
-------------------------------------
Fix Version/s: (was: 0.14.0)
0.15.0
> listing of an output directory shortly after job completion fails
> -----------------------------------------------------------------
>
> Key: HADOOP-1612
> URL: https://issues.apache.org/jira/browse/HADOOP-1612
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.14.0
> Reporter: Christian Kunz
> Assignee: Arun C Murthy
> Fix For: 0.15.0
>
>
> Sometimes, after a job finishes, and another application wants to rename dfs files created by that job, listing of the output directory containing the newly created files fails. File creation and directory listing is done via libhdfs, but it is unlikely that this makes any difference, therefore, I add this to the mapred component.
> It might be a race condition: does the job complete before the files in the output directory are promoted?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1612) listing of an output directory
shortly after job completion fails
Posted by "Christian Kunz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12516620 ]
Christian Kunz commented on HADOOP-1612:
----------------------------------------
I checked the logs of about 150 applications run with the July-25 nightly build which incorporates HADOOP-1576, and also ignoring _${taskid} subdirectories in the output directory:
Outcome:
1) no loss of files (looks like this was fixed by HADOOP-1576, thank you ***)
2) no movement of undesired files
but:
3) listDirectory of output directory still occasionally fails up to 10 seconds after job completion (maybe a DFS issue?)
4) _${taskid} subdirectories not completely cleaned up even days after job completion.
I think the issue should be reopened but not as a blocker.
> listing of an output directory shortly after job completion fails
> -----------------------------------------------------------------
>
> Key: HADOOP-1612
> URL: https://issues.apache.org/jira/browse/HADOOP-1612
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.14.0
> Reporter: Christian Kunz
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.14.0
>
>
> Sometimes, after a job finishes, and another application wants to rename dfs files created by that job, listing of the output directory containing the newly created files fails. File creation and directory listing is done via libhdfs, but it is unlikely that this makes any difference, therefore, I add this to the mapred component.
> It might be a race condition: does the job complete before the files in the output directory are promoted?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1612) listing of an output directory
shortly after job completion fails
Posted by "Christian Kunz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christian Kunz updated HADOOP-1612:
-----------------------------------
Priority: Major (was: Blocker)
> listing of an output directory shortly after job completion fails
> -----------------------------------------------------------------
>
> Key: HADOOP-1612
> URL: https://issues.apache.org/jira/browse/HADOOP-1612
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.14.0
> Reporter: Christian Kunz
> Assignee: Arun C Murthy
> Fix For: 0.14.0
>
>
> Sometimes, after a job finishes, and another application wants to rename dfs files created by that job, listing of the output directory containing the newly created files fails. File creation and directory listing is done via libhdfs, but it is unlikely that this makes any difference, therefore, I add this to the mapred component.
> It might be a race condition: does the job complete before the files in the output directory are promoted?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1612) listing of an output directory
shortly after job completion fails
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12514412 ]
Arun C Murthy commented on HADOOP-1612:
---------------------------------------
Thanks for the info Christian...
bq. From this it looks as if the framework is still active with the output directory even after job completion.
Ah, this is entirely feasible and here is how:
The job is declared as *success* as soon as all the {{TIP}}s are complete (and clearly at this point all the successful sub-task files have been promoted), however there might still be speculative tasks of the completed {{TIP}}s still active at this point. Hence when they are killed (or succeed) the framework just discards their output files (the *_1* directories in your logs)... which are precisely the logs you see.
Of course I'll keep poking and appreciate any info from you...
> listing of an output directory shortly after job completion fails
> -----------------------------------------------------------------
>
> Key: HADOOP-1612
> URL: https://issues.apache.org/jira/browse/HADOOP-1612
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.14.0
> Reporter: Christian Kunz
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.14.0
>
>
> Sometimes, after a job finishes, and another application wants to rename dfs files created by that job, listing of the output directory containing the newly created files fails. File creation and directory listing is done via libhdfs, but it is unlikely that this makes any difference, therefore, I add this to the mapred component.
> It might be a race condition: does the job complete before the files in the output directory are promoted?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1612) listing of an output directory
shortly after job completion fails
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-1612:
----------------------------------
Fix Version/s: 0.14.0
Priority: Blocker (was: Major)
> listing of an output directory shortly after job completion fails
> -----------------------------------------------------------------
>
> Key: HADOOP-1612
> URL: https://issues.apache.org/jira/browse/HADOOP-1612
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.14.0
> Reporter: Christian Kunz
> Priority: Blocker
> Fix For: 0.14.0
>
>
> Sometimes, after a job finishes, and another application wants to rename dfs files created by that job, listing of the output directory containing the newly created files fails. File creation and directory listing is done via libhdfs, but it is unlikely that this makes any difference, therefore, I add this to the mapred component.
> It might be a race condition: does the job complete before the files in the output directory are promoted?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1612) listing of an output directory
shortly after job completion fails
Posted by "Christian Kunz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12514567 ]
Christian Kunz commented on HADOOP-1612:
----------------------------------------
1) Yes, our applications use the temporary task directory ${map.output.dir}/_${taskid} (the jobConf value of mapred.output.dir) for creating output files or directories.
2) No, HADOOP-1576 is not yet part of our release.
> listing of an output directory shortly after job completion fails
> -----------------------------------------------------------------
>
> Key: HADOOP-1612
> URL: https://issues.apache.org/jira/browse/HADOOP-1612
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.14.0
> Reporter: Christian Kunz
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.14.0
>
>
> Sometimes, after a job finishes, and another application wants to rename dfs files created by that job, listing of the output directory containing the newly created files fails. File creation and directory listing is done via libhdfs, but it is unlikely that this makes any difference, therefore, I add this to the mapred component.
> It might be a race condition: does the job complete before the files in the output directory are promoted?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1612) listing of an output directory
shortly after job completion fails
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12514554 ]
Arun C Murthy commented on HADOOP-1612:
---------------------------------------
Christian, another pair of questions:
bq. In many of our applications we use the PIPES interface and create multiple files using libhdfs in the output directory of the task.
I'd like to clarify that you are creating the files (using libhdfs) in the temporary task sub-directory (i.e. ${map.output.dir}/_${taskid}), since only files/dirs there can be safely relied upon to be promoted by the framework.
I'd also like to understand if you are using HADOOP-1576 which contained some minor improvements to processing speculative tasks...
Thanks!
> listing of an output directory shortly after job completion fails
> -----------------------------------------------------------------
>
> Key: HADOOP-1612
> URL: https://issues.apache.org/jira/browse/HADOOP-1612
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.14.0
> Reporter: Christian Kunz
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.14.0
>
>
> Sometimes, after a job finishes, and another application wants to rename dfs files created by that job, listing of the output directory containing the newly created files fails. File creation and directory listing is done via libhdfs, but it is unlikely that this makes any difference, therefore, I add this to the mapred component.
> It might be a race condition: does the job complete before the files in the output directory are promoted?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1612) listing of an output directory
shortly after job completion fails
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12515217 ]
Arun C Murthy commented on HADOOP-1612:
---------------------------------------
Christian, I've spent a fair amount of time trying to reproduce the _lost files_ case without much headway... although the issue with the _${taskid} subdirs turning up is fairly easy to reproduce and as we discussed it is an unfortunate side-effect of speculative tasks killed *after* job completion.
I'll keep trying to see if I get some toe-hold, meanwhile:
a) Could you just ignore the the _${taskid} files while you are moving stuff from your job output dir.
b) Try and incorporate HADOOP-1576, which at the very least, helps in debugging.
Thanks!
> listing of an output directory shortly after job completion fails
> -----------------------------------------------------------------
>
> Key: HADOOP-1612
> URL: https://issues.apache.org/jira/browse/HADOOP-1612
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.14.0
> Reporter: Christian Kunz
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.14.0
>
>
> Sometimes, after a job finishes, and another application wants to rename dfs files created by that job, listing of the output directory containing the newly created files fails. File creation and directory listing is done via libhdfs, but it is unlikely that this makes any difference, therefore, I add this to the mapred component.
> It might be a race condition: does the job complete before the files in the output directory are promoted?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1612) listing of an output directory
shortly after job completion fails
Posted by "Christian Kunz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12532781 ]
Christian Kunz commented on HADOOP-1612:
----------------------------------------
Just wanted to mention that I still see that listing a job output directory using libhdfs after successful job completion sometimes fails for up to 6 secs.
> listing of an output directory shortly after job completion fails
> -----------------------------------------------------------------
>
> Key: HADOOP-1612
> URL: https://issues.apache.org/jira/browse/HADOOP-1612
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.14.0
> Reporter: Christian Kunz
> Assignee: Arun C Murthy
> Fix For: 0.15.0
>
>
> Sometimes, after a job finishes, and another application wants to rename dfs files created by that job, listing of the output directory containing the newly created files fails. File creation and directory listing is done via libhdfs, but it is unlikely that this makes any difference, therefore, I add this to the mapred component.
> It might be a race condition: does the job complete before the files in the output directory are promoted?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1612) listing of an output directory
shortly after job completion fails
Posted by "Christian Kunz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12514363 ]
Christian Kunz commented on HADOOP-1612:
----------------------------------------
In many of our applications we use the PIPES interface and create multiple files using libhdfs in the output directory of the task. We rely on the framework to promote the output directory of the winning task among speculatively executed tasks, and to kill and clean the losing tasks completely.
Failure to list files in the output directory is just one of several symptoms of this issue. We occasionally lose files after moving them from the output directory to some other destination, or we still find files in temporary task subdirectories of the output directory. E.g we see log entries like the following (happened 17 seconds after job completion, this task was a killed speculative task).
NOTICE(06:58:08,3086960320): Rename <outputDir>/_task_0120_r_000004_1/4/<filename> to <destinationFile>
I also see log entries on the NameNode several seconds after job completion looking like
2007-07-19 23:57:53,791 INFO org.apache.hadoop.dfs.StateChange: BLOCK* NameSystem.allocateBlock: <outputDir>/_task_0120_r_000068_1/68/<filename> blk_5152182437135923221 is created and added to pendingCreates and pendingCreateBlocks
>From this it looks as if the framework is still active with the output directory even after job completion.
> listing of an output directory shortly after job completion fails
> -----------------------------------------------------------------
>
> Key: HADOOP-1612
> URL: https://issues.apache.org/jira/browse/HADOOP-1612
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.14.0
> Reporter: Christian Kunz
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.14.0
>
>
> Sometimes, after a job finishes, and another application wants to rename dfs files created by that job, listing of the output directory containing the newly created files fails. File creation and directory listing is done via libhdfs, but it is unlikely that this makes any difference, therefore, I add this to the mapred component.
> It might be a race condition: does the job complete before the files in the output directory are promoted?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.