You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Hairong Kuang (JIRA)" <ji...@apache.org> on 2008/12/01 22:39:44 UTC
[jira] Created: (HADOOP-4746) Job output directory should be
normalized
Job output directory should be normalized
-----------------------------------------
Key: HADOOP-4746
URL: https://issues.apache.org/jira/browse/HADOOP-4746
Project: Hadoop Core
Issue Type: Bug
Affects Versions: 0.18.0
Reporter: Hairong Kuang
Assignee: Hairong Kuang
Fix For: 0.18.3
JobClient should normalize the output directory by calling FileSystem#makeQualified() in order to avoid the problem described in HADOOP-4717.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4746) Job output directory should be
normalized
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-4746:
------------------------------------
Priority: Blocker (was: Major)
> Job output directory should be normalized
> -----------------------------------------
>
> Key: HADOOP-4746
> URL: https://issues.apache.org/jira/browse/HADOOP-4746
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.3
>
> Attachments: outDir.patch, outDir1.patch, outDir2-br18.patch, outDir2.patch
>
>
> JobClient should normalize the output directory by calling FileSystem#makeQualified() in order to avoid the problem described in HADOOP-4717.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4746) Job output directory should be
normalized
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12652214#action_12652214 ]
Arun C Murthy commented on HADOOP-4746:
---------------------------------------
The patch is fine, though ignoring the exception bothers me. Should we atleast log that exception, if not abort?
> Job output directory should be normalized
> -----------------------------------------
>
> Key: HADOOP-4746
> URL: https://issues.apache.org/jira/browse/HADOOP-4746
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.18.3
>
> Attachments: outDir.patch
>
>
> JobClient should normalize the output directory by calling FileSystem#makeQualified() in order to avoid the problem described in HADOOP-4717.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4746) Job output directory should be
normalized
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-4746:
----------------------------------
Attachment: outDir1.patch
This patch has two changes:
1. make FileOutputFormat#setOutputPath to throw an IOException;
2. The output dir could be in a file system which is different from the one defined in jobconf; So use Path.getFileSystem to get the file system handle.
> Job output directory should be normalized
> -----------------------------------------
>
> Key: HADOOP-4746
> URL: https://issues.apache.org/jira/browse/HADOOP-4746
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.18.3
>
> Attachments: outDir.patch, outDir1.patch
>
>
> JobClient should normalize the output directory by calling FileSystem#makeQualified() in order to avoid the problem described in HADOOP-4717.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4746) Job output directory should be
normalized
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-4746:
----------------------------------
Attachment: outDir.patch
> Job output directory should be normalized
> -----------------------------------------
>
> Key: HADOOP-4746
> URL: https://issues.apache.org/jira/browse/HADOOP-4746
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.18.3
>
> Attachments: outDir.patch
>
>
> JobClient should normalize the output directory by calling FileSystem#makeQualified() in order to avoid the problem described in HADOOP-4717.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4746) Job output directory should be
normalized
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-4746:
----------------------------------
Attachment: outDir2.patch
This patch incorporates Owen's comment. It normalize the output directory in JobClient#submitJob. Ideally setInputPath & getInputPath should be methods of JobConf.
> Job output directory should be normalized
> -----------------------------------------
>
> Key: HADOOP-4746
> URL: https://issues.apache.org/jira/browse/HADOOP-4746
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.18.3
>
> Attachments: outDir.patch, outDir1.patch, outDir2.patch
>
>
> JobClient should normalize the output directory by calling FileSystem#makeQualified() in order to avoid the problem described in HADOOP-4717.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4746) Job output directory should be
normalized
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653029#action_12653029 ]
Doug Cutting commented on HADOOP-4746:
--------------------------------------
Might this be done in FileOutputFormat#checkOutputSpecs()? That would better encapsulate this file-specific operation, and it already throws IOException, so it wouldn't break compatibility.
> Ideally setInputPath & getInputPath should be methods of JobConf.
No, we'd like to keep all file-specific stuff in FileOutputFormat, since jobs can take non-file input and output.
> Job output directory should be normalized
> -----------------------------------------
>
> Key: HADOOP-4746
> URL: https://issues.apache.org/jira/browse/HADOOP-4746
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.3
>
> Attachments: outDir.patch, outDir1.patch, outDir2-br18.patch, outDir2.patch
>
>
> JobClient should normalize the output directory by calling FileSystem#makeQualified() in order to avoid the problem described in HADOOP-4717.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4746) Job output directory should be
normalized
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653047#action_12653047 ]
Hairong Kuang commented on HADOOP-4746:
---------------------------------------
The down side of doing this in FileOutputFormat#checkOutputSpecs() is that if a user writes its own customized OutputFormat, the user might forget to normalize the output directory in checkOutputSpecs(). Otherwise I am OK with this suggestion.
> Job output directory should be normalized
> -----------------------------------------
>
> Key: HADOOP-4746
> URL: https://issues.apache.org/jira/browse/HADOOP-4746
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.3
>
> Attachments: outDir.patch, outDir1.patch, outDir2-br18.patch, outDir2.patch
>
>
> JobClient should normalize the output directory by calling FileSystem#makeQualified() in order to avoid the problem described in HADOOP-4717.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4746) Job output directory should be
normalized
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653501#action_12653501 ]
Hairong Kuang commented on HADOOP-4746:
---------------------------------------
ant test-core passed:
BUILD SUCCESSFUL
Total time: 114 minutes 21 seconds
ant test-patch result:
[exec] -1 overall.
[exec] +1 @author. The patch does not contain any @author tags.
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
> Job output directory should be normalized
> -----------------------------------------
>
> Key: HADOOP-4746
> URL: https://issues.apache.org/jira/browse/HADOOP-4746
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.3
>
> Attachments: outDir.patch, outDir1.patch, outDir2-br18.patch, outDir2.patch, outDir3.patch
>
>
> JobClient should normalize the output directory by calling FileSystem#makeQualified() in order to avoid the problem described in HADOOP-4717.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-4746) Job output directory should be
normalized
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang resolved HADOOP-4746.
-----------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
A unit test will be added in HADOOP-4717 to test the new code.
I've just committed this!
> Job output directory should be normalized
> -----------------------------------------
>
> Key: HADOOP-4746
> URL: https://issues.apache.org/jira/browse/HADOOP-4746
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.3
>
> Attachments: outDir.patch, outDir1.patch, outDir2-br18.patch, outDir2.patch, outDir3.patch
>
>
> JobClient should normalize the output directory by calling FileSystem#makeQualified() in order to avoid the problem described in HADOOP-4717.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4746) Job output directory should be
normalized
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-4746:
----------------------------------
Attachment: outDir2-br18.patch
A patch for 0.18.
> Job output directory should be normalized
> -----------------------------------------
>
> Key: HADOOP-4746
> URL: https://issues.apache.org/jira/browse/HADOOP-4746
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.18.3
>
> Attachments: outDir.patch, outDir1.patch, outDir2-br18.patch, outDir2.patch
>
>
> JobClient should normalize the output directory by calling FileSystem#makeQualified() in order to avoid the problem described in HADOOP-4717.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4746) Job output directory should be
normalized
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12652498#action_12652498 ]
Owen O'Malley commented on HADOOP-4746:
---------------------------------------
Rather than make the incompatible API change, I'd propose putting the qualification into JobClient where it is about to be submitted to the JobTracker. I believe the input path is already made absolute there.
> Job output directory should be normalized
> -----------------------------------------
>
> Key: HADOOP-4746
> URL: https://issues.apache.org/jira/browse/HADOOP-4746
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.18.3
>
> Attachments: outDir.patch, outDir1.patch
>
>
> JobClient should normalize the output directory by calling FileSystem#makeQualified() in order to avoid the problem described in HADOOP-4717.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4746) Job output directory should be
normalized
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-4746:
----------------------------------
Attachment: outDir3.patch
Here comes a new patch, which normalizes the output directory in FileOutputFormat#checkOutputSpecs().
> Job output directory should be normalized
> -----------------------------------------
>
> Key: HADOOP-4746
> URL: https://issues.apache.org/jira/browse/HADOOP-4746
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.3
>
> Attachments: outDir.patch, outDir1.patch, outDir2-br18.patch, outDir2.patch, outDir3.patch
>
>
> JobClient should normalize the output directory by calling FileSystem#makeQualified() in order to avoid the problem described in HADOOP-4717.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4746) Job output directory should be
normalized
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654092#action_12654092 ]
Hudson commented on HADOOP-4746:
--------------------------------
Integrated in Hadoop-trunk #680 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/680/])
> Job output directory should be normalized
> -----------------------------------------
>
> Key: HADOOP-4746
> URL: https://issues.apache.org/jira/browse/HADOOP-4746
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.3
>
> Attachments: outDir.patch, outDir1.patch, outDir2-br18.patch, outDir2.patch, outDir3.patch
>
>
> JobClient should normalize the output directory by calling FileSystem#makeQualified() in order to avoid the problem described in HADOOP-4717.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4746) Job output directory should be
normalized
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-4746:
----------------------------------
Component/s: mapred
> Job output directory should be normalized
> -----------------------------------------
>
> Key: HADOOP-4746
> URL: https://issues.apache.org/jira/browse/HADOOP-4746
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.18.3
>
> Attachments: outDir.patch
>
>
> JobClient should normalize the output directory by calling FileSystem#makeQualified() in order to avoid the problem described in HADOOP-4717.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4746) Job output directory should be
normalized
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12653483#action_12653483 ]
Doug Cutting commented on HADOOP-4746:
--------------------------------------
+1 This looks good to me.
> Job output directory should be normalized
> -----------------------------------------
>
> Key: HADOOP-4746
> URL: https://issues.apache.org/jira/browse/HADOOP-4746
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: 0.18.3
>
> Attachments: outDir.patch, outDir1.patch, outDir2-br18.patch, outDir2.patch, outDir3.patch
>
>
> JobClient should normalize the output directory by calling FileSystem#makeQualified() in order to avoid the problem described in HADOOP-4717.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4746) Job output directory should be
normalized
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12652528#action_12652528 ]
Hairong Kuang commented on HADOOP-4746:
---------------------------------------
Output directory is set by the map/reduce application. I am not familiar with JobClient code. But I do not think how it works if we make change in JobClient. Reading it from jobConf, normailizing it, and then seting it again?
> Job output directory should be normalized
> -----------------------------------------
>
> Key: HADOOP-4746
> URL: https://issues.apache.org/jira/browse/HADOOP-4746
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.18.3
>
> Attachments: outDir.patch, outDir1.patch
>
>
> JobClient should normalize the output directory by calling FileSystem#makeQualified() in order to avoid the problem described in HADOOP-4717.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.