You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Jane Chen (Created) (JIRA)" <ji...@apache.org> on 2011/11/08 20:47:51 UTC
[jira] [Created] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Compatibility issue with 0.20.203.
----------------------------------
Key: MAPREDUCE-3377
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.20.203.0
Environment: 0.20.203.0 standalone JDK 1.6 Linux
Reporter: Jane Chen
I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241217#comment-13241217 ]
Hudson commented on MAPREDUCE-3377:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #1034 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1034/])
MAPREDUCE-3377. Added a unit test to ensure OutputCommitter.checkOutputSpecs is called prior to copying job.xml. Contributed by Jane Chen. (Revision 1306736)
Result = FAILURE
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1306736
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMROutputFormat.java
> Compatibility issue with 0.20.203.
> ----------------------------------
>
> Key: MAPREDUCE-3377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.203.0
> Environment: 0.20.203.0 standalone JDK 1.6 Linux
> Reporter: Jane Chen
> Assignee: Jane Chen
> Fix For: 1.0.3, 2.0.0
>
> Attachments: err-after, err-before, mapreduce-3377-branch-1.patch, mapreduce-3377-branch-1.patch, mapreduce-3377.patch, mapreduce-3377.patch, mapreduce-3377.patch, mapreduce-3377.patch, out-after, out-before
>
>
> I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
> I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188290#comment-13188290 ]
Hadoop QA commented on MAPREDUCE-3377:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12510953/mapreduce-3377.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1624//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1624//console
This message is automatically generated.
> Compatibility issue with 0.20.203.
> ----------------------------------
>
> Key: MAPREDUCE-3377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.203.0, 1.0.0
> Environment: 0.20.203.0 standalone JDK 1.6 Linux
> Reporter: Jane Chen
> Assignee: Jane Chen
> Fix For: 1.1.0
>
> Attachments: err-after, err-before, mapreduce-3377-branch-1.patch, mapreduce-3377.patch, mapreduce-3377.patch, mapreduce-3377.patch, out-after, out-before
>
>
> I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
> I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241067#comment-13241067 ]
Hudson commented on MAPREDUCE-3377:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1946 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1946/])
MAPREDUCE-3377. Added a unit test to ensure OutputCommitter.checkOutputSpecs is called prior to copying job.xml. Contributed by Jane Chen. (Revision 1306736)
Result = SUCCESS
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1306736
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMROutputFormat.java
> Compatibility issue with 0.20.203.
> ----------------------------------
>
> Key: MAPREDUCE-3377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.203.0
> Environment: 0.20.203.0 standalone JDK 1.6 Linux
> Reporter: Jane Chen
> Assignee: Jane Chen
> Fix For: 1.0.3, 2.0.0
>
> Attachments: err-after, err-before, mapreduce-3377-branch-1.patch, mapreduce-3377-branch-1.patch, mapreduce-3377.patch, mapreduce-3377.patch, mapreduce-3377.patch, mapreduce-3377.patch, out-after, out-before
>
>
> I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
> I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266972#comment-13266972 ]
Matt Foley commented on MAPREDUCE-3377:
---------------------------------------
The merge to branch-1.0 included, apparently by accident, changes to build.xml that broke the native builds of task-controller and jsvc. I've reverted the build.xml change only. This error only affected branch-1.0, not branch-1 nor the other branches.
> Compatibility issue with 0.20.203.
> ----------------------------------
>
> Key: MAPREDUCE-3377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.203.0
> Environment: 0.20.203.0 standalone JDK 1.6 Linux
> Reporter: Jane Chen
> Assignee: Jane Chen
> Fix For: 1.0.3, 2.0.0
>
> Attachments: err-after, err-before, mapreduce-3377-branch-1.patch, mapreduce-3377-branch-1.patch, mapreduce-3377.patch, mapreduce-3377.patch, mapreduce-3377.patch, mapreduce-3377.patch, out-after, out-before
>
>
> I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
> I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Posted by "Harsh J (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188082#comment-13188082 ]
Harsh J commented on MAPREDUCE-3377:
------------------------------------
This seems to affect only the new API. The way the stable API does this remains consistent with how it used to be in 0.20.2.
Test case does fail without JC change. +1 for this patch, but we need a test case for trunk too, and if it fails without changes, a fix as well.
On 0.22+, the area of change would be under JobSubmitter#checkSpecs(…).
> Compatibility issue with 0.20.203.
> ----------------------------------
>
> Key: MAPREDUCE-3377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.203.0, 1.0.0
> Environment: 0.20.203.0 standalone JDK 1.6 Linux
> Reporter: Jane Chen
> Assignee: Jane Chen
> Fix For: 1.1.0
>
> Attachments: mapreduce-3377-branch-1.patch, mapreduce-3377.patch, mapreduce-3377.patch
>
>
> I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
> I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Posted by "Jane Chen (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jane Chen updated MAPREDUCE-3377:
---------------------------------
Attachment: mapreduce-3377.patch
This worked for me. Can someone please review the attached?
> Compatibility issue with 0.20.203.
> ----------------------------------
>
> Key: MAPREDUCE-3377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.203.0
> Environment: 0.20.203.0 standalone JDK 1.6 Linux
> Reporter: Jane Chen
> Attachments: mapreduce-3377.patch
>
>
> I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
> I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Posted by "Jane Chen (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jane Chen updated MAPREDUCE-3377:
---------------------------------
Attachment: mapreduce-3377.patch
New patch generated on branch 1 with unit test.
> Compatibility issue with 0.20.203.
> ----------------------------------
>
> Key: MAPREDUCE-3377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.203.0, 1.0.0
> Environment: 0.20.203.0 standalone JDK 1.6 Linux
> Reporter: Jane Chen
> Assignee: Jane Chen
> Attachments: mapreduce-3377.patch, mapreduce-3377.patch
>
>
> I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
> I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191824#comment-13191824 ]
Hadoop QA commented on MAPREDUCE-3377:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511607/mapreduce-3377.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1654//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1654//console
This message is automatically generated.
> Compatibility issue with 0.20.203.
> ----------------------------------
>
> Key: MAPREDUCE-3377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.203.0
> Environment: 0.20.203.0 standalone JDK 1.6 Linux
> Reporter: Jane Chen
> Assignee: Jane Chen
> Attachments: err-after, err-before, mapreduce-3377-branch-1.patch, mapreduce-3377-branch-1.patch, mapreduce-3377.patch, mapreduce-3377.patch, mapreduce-3377.patch, mapreduce-3377.patch, out-after, out-before
>
>
> I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
> I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Posted by "Harsh J (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-3377:
-------------------------------
Target Version/s: 0.23.1, 0.24.0, 1.1.0
Affects Version/s: (was: 1.0.0)
Fix Version/s: (was: 1.1.0)
> Compatibility issue with 0.20.203.
> ----------------------------------
>
> Key: MAPREDUCE-3377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.203.0
> Environment: 0.20.203.0 standalone JDK 1.6 Linux
> Reporter: Jane Chen
> Assignee: Jane Chen
> Attachments: err-after, err-before, mapreduce-3377-branch-1.patch, mapreduce-3377.patch, mapreduce-3377.patch, mapreduce-3377.patch, out-after, out-before
>
>
> I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
> I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Posted by "Harsh J (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191610#comment-13191610 ]
Harsh J commented on MAPREDUCE-3377:
------------------------------------
That's fine too.
> Compatibility issue with 0.20.203.
> ----------------------------------
>
> Key: MAPREDUCE-3377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.203.0
> Environment: 0.20.203.0 standalone JDK 1.6 Linux
> Reporter: Jane Chen
> Assignee: Jane Chen
> Attachments: err-after, err-before, mapreduce-3377-branch-1.patch, mapreduce-3377.patch, mapreduce-3377.patch, mapreduce-3377.patch, out-after, out-before
>
>
> I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
> I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Posted by "Arun C Murthy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188049#comment-13188049 ]
Arun C Murthy commented on MAPREDUCE-3377:
------------------------------------------
Jane, currently we can only run the patch against trunk. Please paste the output of 'ant test-patch' here and also ensure 'ant test' passes. Thanks.
Also, could you please provide a patch for trunk alongwith the one for branch-1 if this issue exists there too? Thanks.
> Compatibility issue with 0.20.203.
> ----------------------------------
>
> Key: MAPREDUCE-3377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.203.0, 1.0.0
> Environment: 0.20.203.0 standalone JDK 1.6 Linux
> Reporter: Jane Chen
> Assignee: Jane Chen
> Fix For: 1.1.0
>
> Attachments: mapreduce-3377-branch-1.patch, mapreduce-3377.patch, mapreduce-3377.patch
>
>
> I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
> I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Posted by "Jane Chen (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jane Chen updated MAPREDUCE-3377:
---------------------------------
Attachment: mapreduce-3377.patch
> Compatibility issue with 0.20.203.
> ----------------------------------
>
> Key: MAPREDUCE-3377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.203.0, 1.0.0
> Environment: 0.20.203.0 standalone JDK 1.6 Linux
> Reporter: Jane Chen
> Assignee: Jane Chen
> Fix For: 1.1.0
>
> Attachments: err-after, err-before, mapreduce-3377-branch-1.patch, mapreduce-3377.patch, mapreduce-3377.patch, mapreduce-3377.patch, out-after, out-before
>
>
> I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
> I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Posted by "Jane Chen (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jane Chen updated MAPREDUCE-3377:
---------------------------------
Attachment: mapreduce-3377-branch-1.patch
Update patch name to reflect branch name.
> Compatibility issue with 0.20.203.
> ----------------------------------
>
> Key: MAPREDUCE-3377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.203.0, 1.0.0
> Environment: 0.20.203.0 standalone JDK 1.6 Linux
> Reporter: Jane Chen
> Assignee: Jane Chen
> Attachments: mapreduce-3377-branch-1.patch, mapreduce-3377.patch, mapreduce-3377.patch
>
>
> I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
> I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3377:
-------------------------------------
Resolution: Fixed
Fix Version/s: 2.0.0
1.0.3
Target Version/s: 0.23.1, 0.24.0, 1.1.0 (was: 1.1.0, 0.24.0, 0.23.1)
Status: Resolved (was: Patch Available)
I just committed this. Thanks Jane!
> Compatibility issue with 0.20.203.
> ----------------------------------
>
> Key: MAPREDUCE-3377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.203.0
> Environment: 0.20.203.0 standalone JDK 1.6 Linux
> Reporter: Jane Chen
> Assignee: Jane Chen
> Fix For: 1.0.3, 2.0.0
>
> Attachments: err-after, err-before, mapreduce-3377-branch-1.patch, mapreduce-3377-branch-1.patch, mapreduce-3377.patch, mapreduce-3377.patch, mapreduce-3377.patch, mapreduce-3377.patch, out-after, out-before
>
>
> I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
> I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185213#comment-13185213 ]
Eli Collins commented on MAPREDUCE-3377:
----------------------------------------
Hi Jane,
You change looks good to me. Would you mind adding a test that would fail currently w/o your patch, and would regress if it was removed?
Thanks,
Eli
> Compatibility issue with 0.20.203.
> ----------------------------------
>
> Key: MAPREDUCE-3377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.203.0, 1.0.0
> Environment: 0.20.203.0 standalone JDK 1.6 Linux
> Reporter: Jane Chen
> Attachments: mapreduce-3377.patch
>
>
> I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
> I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241075#comment-13241075 ]
Hudson commented on MAPREDUCE-3377:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1959 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1959/])
MAPREDUCE-3377. Added a unit test to ensure OutputCommitter.checkOutputSpecs is called prior to copying job.xml. Contributed by Jane Chen. (Revision 1306736)
Result = FAILURE
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1306736
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMROutputFormat.java
> Compatibility issue with 0.20.203.
> ----------------------------------
>
> Key: MAPREDUCE-3377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.203.0
> Environment: 0.20.203.0 standalone JDK 1.6 Linux
> Reporter: Jane Chen
> Assignee: Jane Chen
> Fix For: 1.0.3, 2.0.0
>
> Attachments: err-after, err-before, mapreduce-3377-branch-1.patch, mapreduce-3377-branch-1.patch, mapreduce-3377.patch, mapreduce-3377.patch, mapreduce-3377.patch, mapreduce-3377.patch, out-after, out-before
>
>
> I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
> I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241239#comment-13241239 ]
Hudson commented on MAPREDUCE-3377:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #999 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/999/])
MAPREDUCE-3377. Added a unit test to ensure OutputCommitter.checkOutputSpecs is called prior to copying job.xml. Contributed by Jane Chen. (Revision 1306736)
Result = FAILURE
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1306736
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMROutputFormat.java
> Compatibility issue with 0.20.203.
> ----------------------------------
>
> Key: MAPREDUCE-3377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.203.0
> Environment: 0.20.203.0 standalone JDK 1.6 Linux
> Reporter: Jane Chen
> Assignee: Jane Chen
> Fix For: 1.0.3, 2.0.0
>
> Attachments: err-after, err-before, mapreduce-3377-branch-1.patch, mapreduce-3377-branch-1.patch, mapreduce-3377.patch, mapreduce-3377.patch, mapreduce-3377.patch, mapreduce-3377.patch, out-after, out-before
>
>
> I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
> I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187914#comment-13187914 ]
Hadoop QA commented on MAPREDUCE-3377:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12510875/mapreduce-3377-branch-1.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1618//console
This message is automatically generated.
> Compatibility issue with 0.20.203.
> ----------------------------------
>
> Key: MAPREDUCE-3377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.203.0, 1.0.0
> Environment: 0.20.203.0 standalone JDK 1.6 Linux
> Reporter: Jane Chen
> Assignee: Jane Chen
> Fix For: 1.1.0
>
> Attachments: mapreduce-3377-branch-1.patch, mapreduce-3377.patch, mapreduce-3377.patch
>
>
> I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
> I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Posted by "Jane Chen (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jane Chen updated MAPREDUCE-3377:
---------------------------------
Attachment: mapreduce-3377.patch
mapreduce-3377-branch-1.patch
> Compatibility issue with 0.20.203.
> ----------------------------------
>
> Key: MAPREDUCE-3377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.203.0
> Environment: 0.20.203.0 standalone JDK 1.6 Linux
> Reporter: Jane Chen
> Assignee: Jane Chen
> Attachments: err-after, err-before, mapreduce-3377-branch-1.patch, mapreduce-3377-branch-1.patch, mapreduce-3377.patch, mapreduce-3377.patch, mapreduce-3377.patch, mapreduce-3377.patch, out-after, out-before
>
>
> I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
> I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Posted by "Jane Chen (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jane Chen updated MAPREDUCE-3377:
---------------------------------
Affects Version/s: 1.0.0
> Compatibility issue with 0.20.203.
> ----------------------------------
>
> Key: MAPREDUCE-3377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.203.0, 1.0.0
> Environment: 0.20.203.0 standalone JDK 1.6 Linux
> Reporter: Jane Chen
> Attachments: mapreduce-3377.patch
>
>
> I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
> I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Posted by "Jane Chen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191605#comment-13191605 ]
Jane Chen commented on MAPREDUCE-3377:
--------------------------------------
How about TestMROutputFormat, since it isn't related to FileOutputFormat?
> Compatibility issue with 0.20.203.
> ----------------------------------
>
> Key: MAPREDUCE-3377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.203.0
> Environment: 0.20.203.0 standalone JDK 1.6 Linux
> Reporter: Jane Chen
> Assignee: Jane Chen
> Attachments: err-after, err-before, mapreduce-3377-branch-1.patch, mapreduce-3377.patch, mapreduce-3377.patch, mapreduce-3377.patch, out-after, out-before
>
>
> I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
> I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Posted by "Jane Chen (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jane Chen updated MAPREDUCE-3377:
---------------------------------
Attachment: (was: mapreduce-3377.patch)
> Compatibility issue with 0.20.203.
> ----------------------------------
>
> Key: MAPREDUCE-3377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.203.0, 1.0.0
> Environment: 0.20.203.0 standalone JDK 1.6 Linux
> Reporter: Jane Chen
> Assignee: Jane Chen
> Attachments: mapreduce-3377.patch, mapreduce-3377.patch
>
>
> I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
> I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Posted by "Jane Chen (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jane Chen updated MAPREDUCE-3377:
---------------------------------
Fix Version/s: 1.1.0
Status: Patch Available (was: Open)
> Compatibility issue with 0.20.203.
> ----------------------------------
>
> Key: MAPREDUCE-3377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 1.0.0, 0.20.203.0
> Environment: 0.20.203.0 standalone JDK 1.6 Linux
> Reporter: Jane Chen
> Assignee: Jane Chen
> Fix For: 1.1.0
>
> Attachments: mapreduce-3377-branch-1.patch, mapreduce-3377.patch, mapreduce-3377.patch
>
>
> I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
> I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Posted by "Jane Chen (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jane Chen updated MAPREDUCE-3377:
---------------------------------
Attachment: err-after
err-before
out-after
out-before
mapreduce-3377.patch
No new diff introduced by the proposed change on branch-1.
The reported issue does not occur on trunk. Attached patch mapreduce-3377.patch includes only the unit test and is generated on trunk.
The other attachments shows that the added unit test passes after the proposed fix and no other diff is introduced.
out-before: stdout running "ant test" before the change.
err-before: stderr running "ant test" before the change.
out-after: stdout running "ant test" after the change.
err-after: stderr running "ant test" after the change.
> Compatibility issue with 0.20.203.
> ----------------------------------
>
> Key: MAPREDUCE-3377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.203.0, 1.0.0
> Environment: 0.20.203.0 standalone JDK 1.6 Linux
> Reporter: Jane Chen
> Assignee: Jane Chen
> Fix For: 1.1.0
>
> Attachments: err-after, err-before, mapreduce-3377-branch-1.patch, mapreduce-3377.patch, mapreduce-3377.patch, mapreduce-3377.patch, out-after, out-before
>
>
> I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
> I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated MAPREDUCE-3377:
----------------------------------
Target Version/s: 1.0.3, 0.23.1, 0.24.0 (was: 0.23.1, 0.24.0, 1.1.0)
> Compatibility issue with 0.20.203.
> ----------------------------------
>
> Key: MAPREDUCE-3377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.203.0
> Environment: 0.20.203.0 standalone JDK 1.6 Linux
> Reporter: Jane Chen
> Assignee: Jane Chen
> Fix For: 1.0.3, 2.0.0-alpha
>
> Attachments: err-after, err-before, mapreduce-3377-branch-1.patch, mapreduce-3377-branch-1.patch, mapreduce-3377.patch, mapreduce-3377.patch, mapreduce-3377.patch, mapreduce-3377.patch, out-after, out-before
>
>
> I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
> I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Posted by "Eli Collins (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins reassigned MAPREDUCE-3377:
--------------------------------------
Assignee: Jane Chen
> Compatibility issue with 0.20.203.
> ----------------------------------
>
> Key: MAPREDUCE-3377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.203.0, 1.0.0
> Environment: 0.20.203.0 standalone JDK 1.6 Linux
> Reporter: Jane Chen
> Assignee: Jane Chen
> Attachments: mapreduce-3377.patch
>
>
> I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
> I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Posted by "Harsh J (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191599#comment-13191599 ]
Harsh J commented on MAPREDUCE-3377:
------------------------------------
Hello Jane,
The patches are fine. I only have a few small nits, some given state of trunk today:
* For trunk:
** Please move the test into hadoop-mapreduce-client-jobclient project. All tests have recently been moved into that. Sorry for this inconvenience but it seems to have happened very recently, to bring them into the maven fold.
* For both:
** Rename test into something more appropriate, perhaps: {{TestMRFileOutputFormat}} for new API.
** We could do with a test for the stable API as well, though there isn't a problem, it will help prevent regressions. You can add this test in {{TestFileOutputFormat}}, as that already targets stable API.
Thanks for taking the time to report and contribute!
> Compatibility issue with 0.20.203.
> ----------------------------------
>
> Key: MAPREDUCE-3377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.203.0, 1.0.0
> Environment: 0.20.203.0 standalone JDK 1.6 Linux
> Reporter: Jane Chen
> Assignee: Jane Chen
> Fix For: 1.1.0
>
> Attachments: err-after, err-before, mapreduce-3377-branch-1.patch, mapreduce-3377.patch, mapreduce-3377.patch, mapreduce-3377.patch, out-after, out-before
>
>
> I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
> I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241066#comment-13241066 ]
Hudson commented on MAPREDUCE-3377:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2021 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2021/])
MAPREDUCE-3377. Added a unit test to ensure OutputCommitter.checkOutputSpecs is called prior to copying job.xml. Contributed by Jane Chen. (Revision 1306736)
Result = SUCCESS
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1306736
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMROutputFormat.java
> Compatibility issue with 0.20.203.
> ----------------------------------
>
> Key: MAPREDUCE-3377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.203.0
> Environment: 0.20.203.0 standalone JDK 1.6 Linux
> Reporter: Jane Chen
> Assignee: Jane Chen
> Fix For: 1.0.3, 2.0.0
>
> Attachments: err-after, err-before, mapreduce-3377-branch-1.patch, mapreduce-3377-branch-1.patch, mapreduce-3377.patch, mapreduce-3377.patch, mapreduce-3377.patch, mapreduce-3377.patch, out-after, out-before
>
>
> I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
> I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Posted by "Jane Chen (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jane Chen updated MAPREDUCE-3377:
---------------------------------
Attachment: (was: mapreduce-3377.patch)
> Compatibility issue with 0.20.203.
> ----------------------------------
>
> Key: MAPREDUCE-3377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.203.0, 1.0.0
> Environment: 0.20.203.0 standalone JDK 1.6 Linux
> Reporter: Jane Chen
> Assignee: Jane Chen
> Fix For: 1.1.0
>
> Attachments: err-after, err-before, mapreduce-3377-branch-1.patch, mapreduce-3377.patch, mapreduce-3377.patch, mapreduce-3377.patch, out-after, out-before
>
>
> I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
> I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Posted by "Jane Chen (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jane Chen updated MAPREDUCE-3377:
---------------------------------
Attachment: mapreduce-3377.patch
> Compatibility issue with 0.20.203.
> ----------------------------------
>
> Key: MAPREDUCE-3377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.203.0, 1.0.0
> Environment: 0.20.203.0 standalone JDK 1.6 Linux
> Reporter: Jane Chen
> Assignee: Jane Chen
> Attachments: mapreduce-3377.patch, mapreduce-3377.patch
>
>
> I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
> I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3377) Compatibility issue with
0.20.203.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242354#comment-13242354 ]
Hudson commented on MAPREDUCE-3377:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #1035 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1035/])
MAPREDUCE-3377. Added a unit test to ensure OutputCommitter.checkOutputSpecs is called prior to copying job.xml. Contributed by Jane Chen. (Revision 1306736)
Result = SUCCESS
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1306736
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMROutputFormat.java
> Compatibility issue with 0.20.203.
> ----------------------------------
>
> Key: MAPREDUCE-3377
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.203.0
> Environment: 0.20.203.0 standalone JDK 1.6 Linux
> Reporter: Jane Chen
> Assignee: Jane Chen
> Fix For: 1.0.3, 2.0.0
>
> Attachments: err-after, err-before, mapreduce-3377-branch-1.patch, mapreduce-3377-branch-1.patch, mapreduce-3377.patch, mapreduce-3377.patch, mapreduce-3377.patch, mapreduce-3377.patch, out-after, out-before
>
>
> I have an OutputFormat which implements Configurable. I set new config entries to a job configuration during checkOutputSpec() so that the tasks will get the config entries through the job configuration. This works fine in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my OutputFormat still has the configuration set, but the copy a task gets does not have the new entries that are set as part of checkOutputSpec().
> I believe that the problem is with JobClient. The job configuration needs to wait till checkOutputSpec() is returned before being cloned and submitted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira