You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Ahmed Radwan (JIRA)" <ji...@apache.org> on 2011/03/01 23:55:37 UTC
[jira] Created: (MAPREDUCE-2350) LocalJobRunner uses
"mapred.output.committer.class" configuration property to retrive the
OutputCommitter (regardless of wether the old API is used or the new API)
LocalJobRunner uses "mapred.output.committer.class" configuration property to retrive the OutputCommitter (regardless of wether the old API is used or the new API)
-------------------------------------------------------------------------------------------------------------------------------------------------------------------
Key: MAPREDUCE-2350
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2350
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Ahmed Radwan
LocalJobRunner uses the "mapred.output.committer.class" configuration property to retrieve the output committer for the job, which can be different from the Output Committer returned from OutputFormat.getOutputCommitter(TaskAttemptContext context). So, two different output committers can be used in the same job.
See line 324 in org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter outputCommitter = job.getOutputCommitter();
Need to modify this behavior to check if the new or the old API is used, and then return the correct output committer.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2350) LocalJobRunner uses
"mapred.output.committer.class" configuration property to retrive the
OutputCommitter (regardless of whether the old API is used or the new API)
Posted by "Devaraj K (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K updated MAPREDUCE-2350:
---------------------------------
Attachment: MAPREDUCE-2350-1.patch
> LocalJobRunner uses "mapred.output.committer.class" configuration property to retrive the OutputCommitter (regardless of whether the old API is used or the new API)
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2350
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2350
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Ahmed Radwan
> Assignee: Devaraj K
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2350-1.patch, MAPREDUCE-2350.patch
>
>
> LocalJobRunner uses the "mapred.output.committer.class" configuration property to retrieve the output committer for the job, which can be different from the Output Committer returned from OutputFormat.getOutputCommitter(TaskAttemptContext context). So, two different output committers can be used in the same job.
> See line 324 in org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter outputCommitter = job.getOutputCommitter();
> Need to modify this behavior to check if the new or the old API is used, and then return the correct output committer.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2350) LocalJobRunner uses
"mapred.output.committer.class" configuration property to retrive the
OutputCommitter (regardless of whether the old API is used or the new API)
Posted by "Devaraj K (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100170#comment-13100170 ]
Devaraj K commented on MAPREDUCE-2350:
--------------------------------------
This failed testcase and findbugs are not related to this patch.
> LocalJobRunner uses "mapred.output.committer.class" configuration property to retrive the OutputCommitter (regardless of whether the old API is used or the new API)
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2350
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2350
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Ahmed Radwan
> Assignee: Devaraj K
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2350-1.patch, MAPREDUCE-2350.patch
>
>
> LocalJobRunner uses the "mapred.output.committer.class" configuration property to retrieve the output committer for the job, which can be different from the Output Committer returned from OutputFormat.getOutputCommitter(TaskAttemptContext context). So, two different output committers can be used in the same job.
> See line 324 in org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter outputCommitter = job.getOutputCommitter();
> Need to modify this behavior to check if the new or the old API is used, and then return the correct output committer.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2350) LocalJobRunner uses
"mapred.output.committer.class" configuration property to retrive the
OutputCommitter (regardless of whether the old API is used or the new API)
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2350:
-------------------------------------
Status: Open (was: Patch Available)
Sorry, the patch has gone stale. Can you please rebase? Thanks.
> LocalJobRunner uses "mapred.output.committer.class" configuration property to retrive the OutputCommitter (regardless of whether the old API is used or the new API)
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2350
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2350
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Ahmed Radwan
> Assignee: Devaraj K
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2350.patch
>
>
> LocalJobRunner uses the "mapred.output.committer.class" configuration property to retrieve the output committer for the job, which can be different from the Output Committer returned from OutputFormat.getOutputCommitter(TaskAttemptContext context). So, two different output committers can be used in the same job.
> See line 324 in org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter outputCommitter = job.getOutputCommitter();
> Need to modify this behavior to check if the new or the old API is used, and then return the correct output committer.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2350) LocalJobRunner uses
"mapred.output.committer.class" configuration property to retrive the
OutputCommitter (regardless of whether the old API is used or the new API)
Posted by "Devaraj K (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K updated MAPREDUCE-2350:
---------------------------------
Status: Patch Available (was: Open)
Rebased the patch against latest directory layout.
> LocalJobRunner uses "mapred.output.committer.class" configuration property to retrive the OutputCommitter (regardless of whether the old API is used or the new API)
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2350
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2350
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Ahmed Radwan
> Assignee: Devaraj K
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2350-1.patch, MAPREDUCE-2350.patch
>
>
> LocalJobRunner uses the "mapred.output.committer.class" configuration property to retrieve the output committer for the job, which can be different from the Output Committer returned from OutputFormat.getOutputCommitter(TaskAttemptContext context). So, two different output committers can be used in the same job.
> See line 324 in org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter outputCommitter = job.getOutputCommitter();
> Need to modify this behavior to check if the new or the old API is used, and then return the correct output committer.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2350) LocalJobRunner uses
"mapred.output.committer.class" configuration property to retrive the
OutputCommitter (regardless of whether the old API is used or the new API)
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100163#comment-13100163 ]
Hadoop QA commented on MAPREDUCE-2350:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12493562/MAPREDUCE-2350-1.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 14 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.server.nodemanager.TestNMAuditLogger
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/639//console
This message is automatically generated.
> LocalJobRunner uses "mapred.output.committer.class" configuration property to retrive the OutputCommitter (regardless of whether the old API is used or the new API)
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2350
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2350
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Ahmed Radwan
> Assignee: Devaraj K
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2350-1.patch, MAPREDUCE-2350.patch
>
>
> LocalJobRunner uses the "mapred.output.committer.class" configuration property to retrieve the output committer for the job, which can be different from the Output Committer returned from OutputFormat.getOutputCommitter(TaskAttemptContext context). So, two different output committers can be used in the same job.
> See line 324 in org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter outputCommitter = job.getOutputCommitter();
> Need to modify this behavior to check if the new or the old API is used, and then return the correct output committer.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2350) LocalJobRunner uses
"mapred.output.committer.class" configuration property to retrive the
OutputCommitter (regardless of whether the old API is used or the new API)
Posted by "Ahmed Radwan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13078927#comment-13078927 ]
Ahmed Radwan commented on MAPREDUCE-2350:
-----------------------------------------
lgtm. Thanks Devaraj!
> LocalJobRunner uses "mapred.output.committer.class" configuration property to retrive the OutputCommitter (regardless of whether the old API is used or the new API)
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2350
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2350
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Ahmed Radwan
> Assignee: Devaraj K
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2350.patch
>
>
> LocalJobRunner uses the "mapred.output.committer.class" configuration property to retrieve the output committer for the job, which can be different from the Output Committer returned from OutputFormat.getOutputCommitter(TaskAttemptContext context). So, two different output committers can be used in the same job.
> See line 324 in org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter outputCommitter = job.getOutputCommitter();
> Need to modify this behavior to check if the new or the old API is used, and then return the correct output committer.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2350) LocalJobRunner uses
"mapred.output.committer.class" configuration property to retrive the
OutputCommitter (regardless of whether the old API is used or the new API)
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13078844#comment-13078844 ]
Shai Erera commented on MAPREDUCE-2350:
---------------------------------------
Thanks Devaraj for looking into this. Patch looks good to me.
> LocalJobRunner uses "mapred.output.committer.class" configuration property to retrive the OutputCommitter (regardless of whether the old API is used or the new API)
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2350
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2350
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Ahmed Radwan
> Assignee: Devaraj K
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2350.patch
>
>
> LocalJobRunner uses the "mapred.output.committer.class" configuration property to retrieve the output committer for the job, which can be different from the Output Committer returned from OutputFormat.getOutputCommitter(TaskAttemptContext context). So, two different output committers can be used in the same job.
> See line 324 in org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter outputCommitter = job.getOutputCommitter();
> Need to modify this behavior to check if the new or the old API is used, and then return the correct output committer.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2350) LocalJobRunner uses
"mapred.output.committer.class" configuration property to retrive the
OutputCommitter (regardless of whether the old API is used or the new API)
Posted by "Ahmed Radwan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan updated MAPREDUCE-2350:
------------------------------------
Summary: LocalJobRunner uses "mapred.output.committer.class" configuration property to retrive the OutputCommitter (regardless of whether the old API is used or the new API) (was: LocalJobRunner uses "mapred.output.committer.class" configuration property to retrive the OutputCommitter (regardless of wether the old API is used or the new API))
> LocalJobRunner uses "mapred.output.committer.class" configuration property to retrive the OutputCommitter (regardless of whether the old API is used or the new API)
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2350
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2350
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Ahmed Radwan
>
> LocalJobRunner uses the "mapred.output.committer.class" configuration property to retrieve the output committer for the job, which can be different from the Output Committer returned from OutputFormat.getOutputCommitter(TaskAttemptContext context). So, two different output committers can be used in the same job.
> See line 324 in org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter outputCommitter = job.getOutputCommitter();
> Need to modify this behavior to check if the new or the old API is used, and then return the correct output committer.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2350) LocalJobRunner uses
"mapred.output.committer.class" configuration property to retrive the
OutputCommitter (regardless of whether the old API is used or the new API)
Posted by "Devaraj K (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K updated MAPREDUCE-2350:
---------------------------------
Attachment: MAPREDUCE-2350.patch
> LocalJobRunner uses "mapred.output.committer.class" configuration property to retrive the OutputCommitter (regardless of whether the old API is used or the new API)
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2350
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2350
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Ahmed Radwan
> Attachments: MAPREDUCE-2350.patch
>
>
> LocalJobRunner uses the "mapred.output.committer.class" configuration property to retrieve the output committer for the job, which can be different from the Output Committer returned from OutputFormat.getOutputCommitter(TaskAttemptContext context). So, two different output committers can be used in the same job.
> See line 324 in org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter outputCommitter = job.getOutputCommitter();
> Need to modify this behavior to check if the new or the old API is used, and then return the correct output committer.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2350) LocalJobRunner uses
"mapred.output.committer.class" configuration property to retrive the
OutputCommitter (regardless of whether the old API is used or the new API)
Posted by "Devaraj K (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K updated MAPREDUCE-2350:
---------------------------------
Fix Version/s: 0.23.0
Assignee: Devaraj K
Status: Patch Available (was: Open)
Provided patch against trunk.
> LocalJobRunner uses "mapred.output.committer.class" configuration property to retrive the OutputCommitter (regardless of whether the old API is used or the new API)
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2350
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2350
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Ahmed Radwan
> Assignee: Devaraj K
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2350.patch
>
>
> LocalJobRunner uses the "mapred.output.committer.class" configuration property to retrieve the output committer for the job, which can be different from the Output Committer returned from OutputFormat.getOutputCommitter(TaskAttemptContext context). So, two different output committers can be used in the same job.
> See line 324 in org.apache.hadoop.mapred.LocalJobRunner: OutputCommitter outputCommitter = job.getOutputCommitter();
> Need to modify this behavior to check if the new or the old API is used, and then return the correct output committer.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira