[jira] [Commented] (MAPREDUCE-3993) reduce fetch catch clause should catch RTEs as well

["Todd Lipcon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408333#comment-13408333 ] 

Todd Lipcon commented on MAPREDUCE-3993:
----------------------------------------

- Need license header on CompressionUtil.java and its unit test
- You can use {{GenericTestUtils.assertExceptionContains}} for the test assertion
- A few style nits:
-- remove the blank lines between {{@param}} entries
-- add a blank line before the javadoc for {{wrappedRead}}
-- we generally don't bother making constants like {{READ_ERROR}} for exception text - not a big deal, just seemed odd to me.
-- The formatting in the {{wrappedRead}} javadoc is a little strange. I would reformat something like:
{code}
/**
 * Utility wrapper for reading from a {@link InputStream}. If the underlying
 * stream throws an unchecked exception, it is re-thrown as an IOException.
 */
{code}
(i.e use full sentences rather than bullet points for easier readability)
-- The following wrapping is weird -- move {{- bytesRead}} up a line:
{code}
+            n = CompressionUtil.wrappedRead(input, shuffleData, bytesRead,
+                shuffleData.length
+                - bytesRead);
{code}
-- Spurious change for the {{"Failed to discard map-output"}} wrapping.
- CompressionUtil can be abstract or have a private constructor, since it's just static methods

Otherwise looks good.
                
> reduce fetch catch clause should catch RTEs as well
> ---------------------------------------------------
>
>                 Key: MAPREDUCE-3993
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3993
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv1, mrv2
>    Affects Versions: 0.23.1, 1.0.2
>            Reporter: Todd Lipcon
>            Assignee: Karthik Kambatla
>         Attachments: MR-3993_branch1.patch
>
>
> When using a compression codec for intermediate compression, some cases of corrupt data can cause the codec to throw exceptions other than IOException (eg java.lang.InternalError). This will currently cause the whole reduce task to fail, instead of simply treating it like another case of a failed fetch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira