You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Harsh J (JIRA)" <ji...@apache.org> on 2011/06/11 20:55:58 UTC
[jira] [Created] (MAPREDUCE-2584) Check for serializers early, and
give out more information regarding missing serializers
Check for serializers early, and give out more information regarding missing serializers
----------------------------------------------------------------------------------------
Key: MAPREDUCE-2584
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: task
Affects Versions: 0.20.2
Reporter: Harsh J
Assignee: Harsh J
As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048231#comment-13048231 ]
jiraposter@reviews.apache.org commented on MAPREDUCE-2584:
----------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/885/
-----------------------------------------------------------
(Updated 2011-06-12 04:36:10.322990)
Review request for hadoop-mapreduce.
Changes
-------
Fixes per Todd's review.
Summary
-------
As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
This issue covers the MapReduce parts of the improvements (made to MapOutputBuffer and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
This addresses bug MAPREDUCE-2584.
http://issues.apache.org/jira/browse/MAPREDUCE-2584
Diffs (updated)
-----
src/java/org/apache/hadoop/mapred/MapTask.java 21599c2
src/java/org/apache/hadoop/mapreduce/JobSubmitter.java 751d528
src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java 5fa329a
Diff: https://reviews.apache.org/r/885/diff
Testing
-------
Added a test case that expects a failure if no io.serializers are present.
Thanks,
Harsh
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472368#comment-13472368 ]
Tom White commented on MAPREDUCE-2584:
--------------------------------------
This looks like a good check to me. Can you avoid using the @SuppressWarnings annotation?
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Attachments: MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, MAPREDUCE-2584.r5.diff, MAPREDUCE-2584.r6.diff, MAPREDUCE-2584.r7.diff, MAPREDUCE-2584.r7.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508336#comment-13508336 ]
Harsh J commented on MAPREDUCE-2584:
------------------------------------
The failing test does look related:
{code}
java.lang.NullPointerException
at org.apache.hadoop.io.serializer.SerializationFactory.<init>(SerializationFactory.java:58)
at org.apache.hadoop.mapreduce.JobSubmitter.checkSerializerSpecs(JobSubmitter.java:456)
at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:424)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:338)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1437)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:617)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:612)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1437)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:612)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:890)
at org.apache.hadoop.conf.TestNoDefaultsJobConf.testNoDefaults(TestNoDefaultsJobConf.java:82)
{code}
But surprised to see it after HADOOP-8588. Perhaps its a different case, investigating.
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Attachments: MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, MAPREDUCE-2584.r5.diff, MAPREDUCE-2584.r6.diff, MAPREDUCE-2584.r7.diff, MAPREDUCE-2584.r7.diff, MAPREDUCE-2584.r8.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2584) Check for serializers early, and
give out more information regarding missing serializers
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-2584:
-------------------------------
Fix Version/s: 0.23.0
Status: Patch Available (was: Open)
Please review patch at https://reviews.apache.org/r/884/
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13097211#comment-13097211 ]
Hadoop QA commented on MAPREDUCE-2584:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12493044/MAPREDUCE-2584.r5.diff
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//console
This message is automatically generated.
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, MAPREDUCE-2584.r5.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2584) Check for serializers early, and
give out more information regarding missing serializers
Posted by "Harsh J (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-2584:
-------------------------------
Target Version/s: 0.23.1, 0.24.0, 1.1.0
Fix Version/s: (was: 0.24.0)
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Attachments: 0.20-security-MAPREDUCE-2584.r5.diff, MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, MAPREDUCE-2584.r5.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "Harsh J (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171824#comment-13171824 ]
Harsh J commented on MAPREDUCE-2584:
------------------------------------
HADOOP-7328 was committed to 0.23/trunk. This one still awaits. I re-ran the tests and they seem to pass.
I'd also like this committed to branch-0.1 (patch was attached earlier).
Let me know if anything further is required!
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Attachments: 0.20-security-MAPREDUCE-2584.r5.diff, MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, MAPREDUCE-2584.r5.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101998#comment-13101998 ]
Harsh J commented on MAPREDUCE-2584:
------------------------------------
Todd,
Looks like some form of build issue.
Here's what I tried to do to get this test to pass:
hadoop $ (Applied patch)
hadoop $ mvn install -DskipTests -P-cbuild
(Success)
hadoop-mapreduce-project $ ant veryclean
(Success)
hadoop-mapreduce-project $ ant -Dresolvers=internal -Dtestcase=TestMRJobClient test
(Failed)
hadoop-mapreduce-project $ grep "find a serializer for the" build/ -R
Binary file build/test/classes/org/apache/hadoop/mapreduce/TestMRJobClient.class matches
Binary file build/test/mapred/classes/org/apache/hadoop/mapreduce/TestMRJobClient.class matches
This looks wrong. That string ought to be present in the binary MR jar as well, but it doesn't show up as part of grep result. It would mean that locally published jars aren't being used? Also, TestMRJobClient lies outside of the client test suites, in the older area… Shouldn't it be moved in?
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, MAPREDUCE-2584.r5.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508585#comment-13508585 ]
Harsh J commented on MAPREDUCE-2584:
------------------------------------
Problem is coming from the fact that when we lookup the IO_SERIALIZATIONS_KEY, we don't pass a default classes set.
We could add that in, but it may get harder to maintain it if we extend the list. In any case, I'll upload another patch.
{code}
java.io.IOException: Couldn't find a serializer for the Map-Output Key class: 'class org.apache.hadoop.io.LongWritable'. If custom serialization is being used, ensure that the 'io.serializations' property is appropriately configured for the Job.
at org.apache.hadoop.mapreduce.JobSubmitter.checkSerializerSpecs(JobSubmitter.java:462)
at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:424)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:338)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1437)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:617)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:612)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1437)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:612)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:890)
at org.apache.hadoop.conf.TestNoDefaultsJobConf.testNoDefaults(TestNoDefaultsJobConf.java:82)
{code}
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Attachments: MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, MAPREDUCE-2584.r5.diff, MAPREDUCE-2584.r6.diff, MAPREDUCE-2584.r7.diff, MAPREDUCE-2584.r7.diff, MAPREDUCE-2584.r8.diff, MAPREDUCE-2584.r9.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047994#comment-13047994 ]
jiraposter@reviews.apache.org commented on MAPREDUCE-2584:
----------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/885/
-----------------------------------------------------------
Review request for hadoop-mapreduce.
Summary
-------
As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
This issue covers the MapReduce parts of the improvements (made to MapOutputBuffer and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
This addresses bug MAPREDUCE-2584.
http://issues.apache.org/jira/browse/MAPREDUCE-2584
Diffs
-----
src/java/org/apache/hadoop/mapred/MapTask.java 21599c2
src/java/org/apache/hadoop/mapreduce/JobSubmitter.java 751d528
src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java 5fa329a
Diff: https://reviews.apache.org/r/885/diff
Testing
-------
Added a test case that expects a failure if no io.serializers are present.
Thanks,
Harsh
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2584) Check for serializers early, and
give out more information regarding missing serializers
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-2584:
-------------------------------
Attachment: MAPREDUCE-2584.r3.diff
Review patch attached.
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2584.r3.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050672#comment-13050672 ]
Hadoop QA commented on MAPREDUCE-2584:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12482789/MAPREDUCE-2584.r2.diff
against trunk revision 1136261.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The patch appears to cause tar ant target to fail.
-1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these core unit tests:
-1 contrib tests. The patch failed contrib unit tests.
-1 system test framework. The patch failed system test framework compile.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/399//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/399//console
This message is automatically generated.
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2584.r2.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471250#comment-13471250 ]
Hadoop QA commented on MAPREDUCE-2584:
--------------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12548162/MAPREDUCE-2584.r7.diff
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.conf.TestNoDefaultsJobConf
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2914//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2914//console
This message is automatically generated.
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Attachments: MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, MAPREDUCE-2584.r5.diff, MAPREDUCE-2584.r6.diff, MAPREDUCE-2584.r7.diff, MAPREDUCE-2584.r7.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508330#comment-13508330 ]
Hadoop QA commented on MAPREDUCE-2584:
--------------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12555676/MAPREDUCE-2584.r8.diff
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.conf.TestNoDefaultsJobConf
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3089//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3089//console
This message is automatically generated.
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Attachments: MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, MAPREDUCE-2584.r5.diff, MAPREDUCE-2584.r6.diff, MAPREDUCE-2584.r7.diff, MAPREDUCE-2584.r7.diff, MAPREDUCE-2584.r8.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508507#comment-13508507 ]
Hadoop QA commented on MAPREDUCE-2584:
--------------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12555708/MAPREDUCE-2584.r9.diff
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.conf.TestNoDefaultsJobConf
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3093//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3093//console
This message is automatically generated.
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Attachments: MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, MAPREDUCE-2584.r5.diff, MAPREDUCE-2584.r6.diff, MAPREDUCE-2584.r7.diff, MAPREDUCE-2584.r7.diff, MAPREDUCE-2584.r8.diff, MAPREDUCE-2584.r9.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "Harsh J (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171825#comment-13171825 ]
Harsh J commented on MAPREDUCE-2584:
------------------------------------
(Er, 1.0)
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Attachments: 0.20-security-MAPREDUCE-2584.r5.diff, MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, MAPREDUCE-2584.r5.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2584) Check for serializers early, and
give out more information regarding missing serializers
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-2584:
-------------------------------
Attachment: MAPREDUCE-2584.r7.diff
Rebased yet again. Part of this patch's changes on the common side were already committed, so am inclined to commit this in as well.
But I'd like to know if there are any concerns with this new front-end check. Anyone? Is throwing an exception bad if there's a ClassNotFound? Should we just WARN?
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Attachments: MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, MAPREDUCE-2584.r5.diff, MAPREDUCE-2584.r6.diff, MAPREDUCE-2584.r7.diff, MAPREDUCE-2584.r7.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101585#comment-13101585 ]
Todd Lipcon commented on MAPREDUCE-2584:
----------------------------------------
Actually, not committing this yet - for some reason the test does not pass on my setup. Maybe something related to Maven/Ivy not pulling the right dependencies... looking into it.
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, MAPREDUCE-2584.r5.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101572#comment-13101572 ]
Todd Lipcon commented on MAPREDUCE-2584:
----------------------------------------
+1
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, MAPREDUCE-2584.r5.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2584) Check for serializers early, and
give out more information regarding missing serializers
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-2584:
-------------------------------
Status: Open (was: Patch Available)
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2584) Check for serializers early, and
give out more information regarding missing serializers
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-2584:
-------------------------------
Attachment: MAPREDUCE-2584.r2.diff
Patch updated as per HADOOP-7328 changes.
(Still blocked by HADOOP-7328 inclusion and will not compile until)
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2584.r2.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2584) Check for serializers early, and
give out more information regarding missing serializers
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-2584:
-------------------------------
Attachment: MAPREDUCE-2584.r7.diff
Rebased to trunk again.
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Attachments: MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, MAPREDUCE-2584.r5.diff, MAPREDUCE-2584.r6.diff, MAPREDUCE-2584.r7.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13055533#comment-13055533 ]
Hadoop QA commented on MAPREDUCE-2584:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12483921/MAPREDUCE-2584.r4.diff
against trunk revision 1139400.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.cli.TestMRCLI
org.apache.hadoop.fs.TestFileSystem
org.apache.hadoop.mapred.TestJobClient
org.apache.hadoop.mapred.TestMiniMRClasspath
org.apache.hadoop.mapreduce.TestMRJobClient
-1 contrib tests. The patch failed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/428//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/428//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/428//console
This message is automatically generated.
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048323#comment-13048323 ]
jiraposter@reviews.apache.org commented on MAPREDUCE-2584:
----------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/885/
-----------------------------------------------------------
(Updated 2011-06-12 14:00:13.155848)
Review request for hadoop-mapreduce.
Changes
-------
Merged exception handling with common, formattable message.
Summary
-------
As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
This issue covers the MapReduce parts of the improvements (made to MapOutputBuffer and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
This addresses bug MAPREDUCE-2584.
http://issues.apache.org/jira/browse/MAPREDUCE-2584
Diffs (updated)
-----
src/java/org/apache/hadoop/mapred/MapTask.java 21599c2
src/java/org/apache/hadoop/mapreduce/JobSubmitter.java 751d528
src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java 5fa329a
Diff: https://reviews.apache.org/r/885/diff
Testing
-------
Added a test case that expects a failure if no io.serializers are present.
Thanks,
Harsh
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2584) Check for serializers early, and
give out more information regarding missing serializers
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-2584:
-------------------------------
Attachment: MAPREDUCE-2584.r6.diff
Rebased patch for current trunk (had to move the test file around).
Added test fails without fix, passes with fix:
{code}
Running org.apache.hadoop.mapreduce.TestMRJobClient
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.03 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1
{code}
Does anyone have any further comments on this improvement? I believe its complete and ought to have gone in with HADOOP-7328 but has instead lied stale for months now, although HADOOP-7328 (a dependency) was put in.
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Attachments: MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, MAPREDUCE-2584.r5.diff, MAPREDUCE-2584.r6.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2584) Check for serializers early, and
give out more information regarding missing serializers
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-2584:
-------------------------------
Attachment: MAPREDUCE-2584.r8.diff
Thanks Tom! I've removed those in this new rebased patch.
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Attachments: MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, MAPREDUCE-2584.r5.diff, MAPREDUCE-2584.r6.diff, MAPREDUCE-2584.r7.diff, MAPREDUCE-2584.r7.diff, MAPREDUCE-2584.r8.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101956#comment-13101956 ]
Harsh J commented on MAPREDUCE-2584:
------------------------------------
Hey Todd,
I'll run the test now. But this requires the HADOOP-7328 patch, to pass (minor return behavior change, being the reason).
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, MAPREDUCE-2584.r5.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048477#comment-13048477 ]
Harsh J commented on MAPREDUCE-2584:
------------------------------------
Test will fail without https://issues.apache.org/jira/browse/HADOOP-7328 as that blocks this.
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2584.r3.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067723#comment-13067723 ]
Harsh J commented on MAPREDUCE-2584:
------------------------------------
This can run once again after HADOOP-7328 is committed.
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048327#comment-13048327 ]
jiraposter@reviews.apache.org commented on MAPREDUCE-2584:
----------------------------------------------------------
bq. On 2011-06-12 04:39:03, Todd Lipcon wrote:
bq. > src/java/org/apache/hadoop/mapreduce/JobSubmitter.java, line 471
bq. > <https://reviews.apache.org/r/885/diff/2/?file=21008#file21008line471>
bq. >
bq. > ah, I noticed one more thing. This catch clause is only for the exceptions you've thrown above.
bq. >
bq. > Rather than do the two nested exceptions, better to have just one layer of exceptions. You could move the generic instructions from the lower exception to a constant and then just append it to either of the two more specific methods above.
Fixed. Unable to run tests right now locally due to a missing protobuf class (which is fixed, but I don't seem to have the fresh jar pulled yet in my maven2 cache). Will update the test when it works (and if it does not).
- Harsh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/885/#review808
-----------------------------------------------------------
On 2011-06-12 14:00:13, Harsh J wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/885/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-06-12 14:00:13)
bq.
bq.
bq. Review request for hadoop-mapreduce.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
bq.
bq. This issue covers the MapReduce parts of the improvements (made to MapOutputBuffer and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
bq.
bq.
bq. This addresses bug MAPREDUCE-2584.
bq. http://issues.apache.org/jira/browse/MAPREDUCE-2584
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/java/org/apache/hadoop/mapred/MapTask.java 21599c2
bq. src/java/org/apache/hadoop/mapreduce/JobSubmitter.java 751d528
bq. src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java 5fa329a
bq.
bq. Diff: https://reviews.apache.org/r/885/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Added a test case that expects a failure if no io.serializers are present.
bq.
bq.
bq. Thanks,
bq.
bq. Harsh
bq.
bq.
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2584) Check for serializers early, and
give out more information regarding missing serializers
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-2584:
-------------------------------
Attachment: MAPREDUCE-2584.r5.diff
Updating patch for trunk…
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, MAPREDUCE-2584.r5.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050380#comment-13050380 ]
jiraposter@reviews.apache.org commented on MAPREDUCE-2584:
----------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/885/
-----------------------------------------------------------
(Updated 2011-06-16 12:24:09.519284)
Review request for hadoop-mapreduce.
Changes
-------
IOException catching ('stead of null).
Summary
-------
As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
This issue covers the MapReduce parts of the improvements (made to MapOutputBuffer and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
This addresses bug MAPREDUCE-2584.
http://issues.apache.org/jira/browse/MAPREDUCE-2584
Diffs (updated)
-----
src/java/org/apache/hadoop/mapreduce/JobSubmitter.java 751d528
src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java 5fa329a
Diff: https://reviews.apache.org/r/885/diff
Testing
-------
Added a test case that expects a failure if no io.serializers are present.
Thanks,
Harsh
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2584.r3.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2584) Check for serializers early, and
give out more information regarding missing serializers
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-2584:
-------------------------------
Status: Open (was: Patch Available)
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Attachments: MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, MAPREDUCE-2584.r5.diff, MAPREDUCE-2584.r6.diff, MAPREDUCE-2584.r7.diff, MAPREDUCE-2584.r7.diff, MAPREDUCE-2584.r8.diff, MAPREDUCE-2584.r9.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048200#comment-13048200 ]
jiraposter@reviews.apache.org commented on MAPREDUCE-2584:
----------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/885/#review806
-----------------------------------------------------------
src/java/org/apache/hadoop/mapreduce/JobSubmitter.java
<https://reviews.apache.org/r/885/#comment1758>
nit: the javadoc style guide says this should be:
@param job the job to check for
(so it formats nicely)
src/java/org/apache/hadoop/mapreduce/JobSubmitter.java
<https://reviews.apache.org/r/885/#comment1759>
I'm generally against putting empty @throws in JavaDoc - it doesn't tell you anything the method signature doesn't already say. I'd just delete these lines
src/java/org/apache/hadoop/mapreduce/JobSubmitter.java
<https://reviews.apache.org/r/885/#comment1760>
same comments apply
src/java/org/apache/hadoop/mapreduce/JobSubmitter.java
<https://reviews.apache.org/r/885/#comment1761>
rather than this "note" why not just change the above line to "Checks if the map-output key and value classes have serializers and deserializers associated with them."?
src/java/org/apache/hadoop/mapreduce/JobSubmitter.java
<https://reviews.apache.org/r/885/#comment1762>
missing a space in this exception method.
src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java
<https://reviews.apache.org/r/885/#comment1763>
add an assert that e.getMessage().contains("Couldn't find a serializer") perhaps?
- Todd
On 2011-06-11 22:16:24, Harsh Chouraria wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/885/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-06-11 22:16:24)
bq.
bq.
bq. Review request for hadoop-mapreduce.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
bq.
bq. This issue covers the MapReduce parts of the improvements (made to MapOutputBuffer and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
bq.
bq.
bq. This addresses bug MAPREDUCE-2584.
bq. http://issues.apache.org/jira/browse/MAPREDUCE-2584
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/java/org/apache/hadoop/mapred/MapTask.java 21599c2
bq. src/java/org/apache/hadoop/mapreduce/JobSubmitter.java 751d528
bq. src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java 5fa329a
bq.
bq. Diff: https://reviews.apache.org/r/885/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Added a test case that expects a failure if no io.serializers are present.
bq.
bq.
bq. Thanks,
bq.
bq. Harsh
bq.
bq.
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048232#comment-13048232 ]
jiraposter@reviews.apache.org commented on MAPREDUCE-2584:
----------------------------------------------------------
bq. On 2011-06-12 02:31:03, Todd Lipcon wrote:
bq. > src/java/org/apache/hadoop/mapreduce/JobSubmitter.java, line 400
bq. > <https://reviews.apache.org/r/885/diff/1/?file=21001#file21001line400>
bq. >
bq. > nit: the javadoc style guide says this should be:
bq. > @param job the job to check for
bq. >
bq. > (so it formats nicely)
Done.
bq. On 2011-06-12 02:31:03, Todd Lipcon wrote:
bq. > src/java/org/apache/hadoop/mapreduce/JobSubmitter.java, lines 401-403
bq. > <https://reviews.apache.org/r/885/diff/1/?file=21001#file21001line401>
bq. >
bq. > I'm generally against putting empty @throws in JavaDoc - it doesn't tell you anything the method signature doesn't already say. I'd just delete these lines
Agreed. It came on by default in Eclipse, and I had let them be instead of filling it up. Removed.
bq. On 2011-06-12 02:31:03, Todd Lipcon wrote:
bq. > src/java/org/apache/hadoop/mapreduce/JobSubmitter.java, lines 417-420
bq. > <https://reviews.apache.org/r/885/diff/1/?file=21001#file21001line417>
bq. >
bq. > same comments apply
Removed.
bq. On 2011-06-12 02:31:03, Todd Lipcon wrote:
bq. > src/java/org/apache/hadoop/mapreduce/JobSubmitter.java, lines 439-440
bq. > <https://reviews.apache.org/r/885/diff/1/?file=21001#file21001line439>
bq. >
bq. > rather than this "note" why not just change the above line to "Checks if the map-output key and value classes have serializers and deserializers associated with them."?
Fixed to a more direct message. Sometimes my commenting language is pretty verbose and confusing, thanks for correcting!
bq. On 2011-06-12 02:31:03, Todd Lipcon wrote:
bq. > src/java/org/apache/hadoop/mapreduce/JobSubmitter.java, line 470
bq. > <https://reviews.apache.org/r/885/diff/1/?file=21001#file21001line470>
bq. >
bq. > missing a space in this exception method.
Added.
bq. On 2011-06-12 02:31:03, Todd Lipcon wrote:
bq. > src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java, line 87
bq. > <https://reviews.apache.org/r/885/diff/1/?file=21002#file21002line87>
bq. >
bq. > add an assert that e.getMessage().contains("Couldn't find a serializer") perhaps?
Added as well.
- Harsh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/885/#review806
-----------------------------------------------------------
On 2011-06-11 22:16:24, Harsh Chouraria wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/885/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-06-11 22:16:24)
bq.
bq.
bq. Review request for hadoop-mapreduce.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
bq.
bq. This issue covers the MapReduce parts of the improvements (made to MapOutputBuffer and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
bq.
bq.
bq. This addresses bug MAPREDUCE-2584.
bq. http://issues.apache.org/jira/browse/MAPREDUCE-2584
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/java/org/apache/hadoop/mapred/MapTask.java 21599c2
bq. src/java/org/apache/hadoop/mapreduce/JobSubmitter.java 751d528
bq. src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java 5fa329a
bq.
bq. Diff: https://reviews.apache.org/r/885/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Added a test case that expects a failure if no io.serializers are present.
bq.
bq.
bq. Thanks,
bq.
bq. Harsh
bq.
bq.
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2584) Check for serializers early, and
give out more information regarding missing serializers
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-2584:
-------------------------------
Status: Patch Available (was: Open)
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2584) Check for serializers early, and
give out more information regarding missing serializers
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-2584:
-------------------------------
Attachment: MAPREDUCE-2584.r3.diff
(Re)attaching an older revision patch, as reviewed (diff #3) on the reviews site by Todd.
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2584) Check for serializers early, and
give out more information regarding missing serializers
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-2584:
-------------------------------------------
Target Version/s: 2.0.0, 3.0.0 (was: 0.24.0, 0.23.1)
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Attachments: 0.20-security-MAPREDUCE-2584.r5.diff, MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, MAPREDUCE-2584.r5.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13055180#comment-13055180 ]
Hadoop QA commented on MAPREDUCE-2584:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12483866/MAPREDUCE-2584.r3.diff
against trunk revision 1139400.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.cli.TestMRCLI
org.apache.hadoop.fs.TestFileSystem
org.apache.hadoop.mapred.TestJobClient
org.apache.hadoop.mapred.TestMiniMRClasspath
org.apache.hadoop.mapreduce.TestMRJobClient
-1 contrib tests. The patch failed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/422//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/422//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/422//console
This message is automatically generated.
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2584) Check for serializers early, and
give out more information regarding missing serializers
Posted by "Harsh J (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-2584:
-------------------------------
Target Version/s: 0.23.1, 0.24.0 (was: 1.1.0, 0.24.0, 0.23.1)
Skipping 1.x, not very important there. Lets focus on getting this into 0.23.x though -- any further comments on latest patch, anyone?
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Attachments: 0.20-security-MAPREDUCE-2584.r5.diff, MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, MAPREDUCE-2584.r5.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2584) Check for serializers early, and
give out more information regarding missing serializers
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-2584:
-------------------------------
Status: Open (was: Patch Available)
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2584.r3.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2584) Check for serializers early, and
give out more information regarding missing serializers
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-2584:
-------------------------------
Attachment: 0.20-security-MAPREDUCE-2584.r5.diff
0.20-security branch patch. Added tests and TestMapRed pass, after applied atop HADOOP-7328's 0.20-security patch.
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
> Attachments: 0.20-security-MAPREDUCE-2584.r5.diff, MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, MAPREDUCE-2584.r5.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048454#comment-13048454 ]
jiraposter@reviews.apache.org commented on MAPREDUCE-2584:
----------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/885/#review811
-----------------------------------------------------------
Looks good. Can you upload latest patch to JIRA so QA-bot picks it up?
- Todd
On 2011-06-12 14:00:13, Harsh J wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/885/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-06-12 14:00:13)
bq.
bq.
bq. Review request for hadoop-mapreduce.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
bq.
bq. This issue covers the MapReduce parts of the improvements (made to MapOutputBuffer and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
bq.
bq.
bq. This addresses bug MAPREDUCE-2584.
bq. http://issues.apache.org/jira/browse/MAPREDUCE-2584
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/java/org/apache/hadoop/mapred/MapTask.java 21599c2
bq. src/java/org/apache/hadoop/mapreduce/JobSubmitter.java 751d528
bq. src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java 5fa329a
bq.
bq. Diff: https://reviews.apache.org/r/885/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Added a test case that expects a failure if no io.serializers are present.
bq.
bq.
bq. Thanks,
bq.
bq. Harsh
bq.
bq.
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2584) Check for serializers early, and
give out more information regarding missing serializers
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-2584:
-------------------------------
Attachment: (was: MAPREDUCE-2584.r3.diff)
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2584.r2.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2584) Check for serializers early, and
give out more information regarding missing serializers
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-2584:
-------------------------------
Attachment: MAPREDUCE-2584.r4.diff
-4 findbugs. My bad, sorry.
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2584) Check for serializers early, and
give out more information regarding missing serializers
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-2584:
-------------------------------
Release Note: Check for proper Key/Value serializers during submission of MR jobs.
Status: Patch Available (was: Open)
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2584.r3.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049195#comment-13049195 ]
Harsh J commented on MAPREDUCE-2584:
------------------------------------
{{From Lists, think its more appropriate here}}
On Tue, Jun 14, 2011 at 6:19 PM, Sudharsan Sampath
<sudharsan.sampath-at-ericsson-dot-com> wrote:
> Just my thoughts...
>
> To me throwing a specific exception would be better. The checkSerializerSpecs attempts to see if we can initialise a new instance of the serializer/deserializer from the jvm where the job is submitted. How does it guarantee that the libs/jars from which these classes are loaded are available for the jvm that executes the job or vice versa? Even if this methods succeeds isn't there a chance then the original problem might occur due to the libs missing from the actual jvm that executes the job?
What you describe about the JVM can surely occur and I've pushed some handling at the MapOutputBuffer level as well but in any case its better to catch it as early as possible.
Don't think guaranteeing that the client is indeed packing things along with the job for the DC is practically possible, but at least it is an improvement to where the client follows a bit of general guidelines I think?
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2584.r3.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404561#comment-13404561 ]
Hadoop QA commented on MAPREDUCE-2584:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12534111/MAPREDUCE-2584.r7.diff
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2536//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2536//console
This message is automatically generated.
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Attachments: MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, MAPREDUCE-2584.r5.diff, MAPREDUCE-2584.r6.diff, MAPREDUCE-2584.r7.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049595#comment-13049595 ]
Hadoop QA commented on MAPREDUCE-2584:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12482324/MAPREDUCE-2584.r3.diff
against trunk revision 1135462.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.cli.TestMRCLI
org.apache.hadoop.fs.TestFileSystem
org.apache.hadoop.mapred.TestJobClient
org.apache.hadoop.mapred.TestMapredSystemDir
org.apache.hadoop.mapred.TestMiniMRClasspath
org.apache.hadoop.mapreduce.TestMRJobClient
-1 contrib tests. The patch failed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/390//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/390//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/390//console
This message is automatically generated.
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2584.r3.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273941#comment-13273941 ]
Hadoop QA commented on MAPREDUCE-2584:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12526621/MAPREDUCE-2584.r6.diff
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2381//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2381//console
This message is automatically generated.
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Attachments: MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, MAPREDUCE-2584.r5.diff, MAPREDUCE-2584.r6.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102137#comment-13102137 ]
Hadoop QA commented on MAPREDUCE-2584:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12493922/0.20-security-MAPREDUCE-2584.r5.diff
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/688//console
This message is automatically generated.
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
> Attachments: 0.20-security-MAPREDUCE-2584.r5.diff, MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, MAPREDUCE-2584.r5.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2584) Check for serializers early, and
give out more information regarding missing serializers
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-2584:
-------------------------------
Attachment: (was: 0.20-security-MAPREDUCE-2584.r5.diff)
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Attachments: MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, MAPREDUCE-2584.r5.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048233#comment-13048233 ]
jiraposter@reviews.apache.org commented on MAPREDUCE-2584:
----------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/885/#review808
-----------------------------------------------------------
src/java/org/apache/hadoop/mapreduce/JobSubmitter.java
<https://reviews.apache.org/r/885/#comment1770>
ah, I noticed one more thing. This catch clause is only for the exceptions you've thrown above.
Rather than do the two nested exceptions, better to have just one layer of exceptions. You could move the generic instructions from the lower exception to a constant and then just append it to either of the two more specific methods above.
- Todd
On 2011-06-12 04:36:10, Harsh J wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/885/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-06-12 04:36:10)
bq.
bq.
bq. Review request for hadoop-mapreduce.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
bq.
bq. This issue covers the MapReduce parts of the improvements (made to MapOutputBuffer and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
bq.
bq.
bq. This addresses bug MAPREDUCE-2584.
bq. http://issues.apache.org/jira/browse/MAPREDUCE-2584
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/java/org/apache/hadoop/mapred/MapTask.java 21599c2
bq. src/java/org/apache/hadoop/mapreduce/JobSubmitter.java 751d528
bq. src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java 5fa329a
bq.
bq. Diff: https://reviews.apache.org/r/885/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Added a test case that expects a failure if no io.serializers are present.
bq.
bq.
bq. Thanks,
bq.
bq. Harsh
bq.
bq.
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2584) Check for serializers early, and
give out more information regarding missing serializers
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-2584:
-------------------------------
Attachment: MAPREDUCE-2584.r9.diff
Patch that addresses the TestNoJobConfDefaults test failure by fixing the SerializationFactory logic, wherein a null may be received for a serializations listing.
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Attachments: MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, MAPREDUCE-2584.r5.diff, MAPREDUCE-2584.r6.diff, MAPREDUCE-2584.r7.diff, MAPREDUCE-2584.r7.diff, MAPREDUCE-2584.r8.diff, MAPREDUCE-2584.r9.diff
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2584) Check for serializers early,
and give out more information regarding missing serializers
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047993#comment-13047993 ]
Harsh J commented on MAPREDUCE-2584:
------------------------------------
Sorry, that should be https://reviews.apache.org/r/885/
> Check for serializers early, and give out more information regarding missing serializers
> ----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2584
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.20.2
> Reporter: Harsh J
> Assignee: Harsh J
> Labels: serializers, tasks
> Fix For: 0.23.0
>
>
> As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..
> This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira