You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Krishna Ramachandran (JIRA)" <ji...@apache.org> on 2010/07/07 02:13:50 UTC
[jira] Created: (MAPREDUCE-1921) IOExceptions should contain the
filename of the broken input files
IOExceptions should contain the filename of the broken input files
------------------------------------------------------------------
Key: MAPREDUCE-1921
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1921
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Krishna Ramachandran
Assignee: Krishna Ramachandran
If bzip or other decompression fails, the IOException does not contain the name of the broken file that caused the exception.
It would be nice if such actions could be avoided in the future by having the name of the files that are broken spelled
out in the exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1921) IOExceptions should contain the
filename of the broken input files
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895116#action_12895116 ]
Krishna Ramachandran commented on MAPREDUCE-1921:
-------------------------------------------------
Thx Dick I will take a look and revise suitably or explain
Regards
Krishna
On 8/3/10 5:16 PM, "Dick King (JIRA)" <ji...@apache.org> wrote:
[ https://issues.apache.org/jira/browse/MAPREDUCE-1921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895096#action_12895096 ]
Dick King commented on MAPREDUCE-1921:
--------------------------------------
{{MapTask.java}}
When creating a new exception to re-throw, perhaps we should duplicate the original exception's type? I realize that there's a lot of code that doesn't do that, but swallowing this information feels wrong. Especially in the case of IO exceptions, I can envision a lot of code that wants to treat certain sub-exceptions a bit differently; some IO exceptions are the caller's fault but some represent oddities that happened in the file system.
Perhaps something like
{noformat}
throw (IOException)(ioe.getClass().getConstructor(String.class, ioe.getClass())
.newInstance("IO error ..." + ... , ioe));
{noformat}
?
This cliche should occur in two places; perhaps we should pull it out and put it in one of the utilities classes? Perhaps IO utilities, since the main use case where I can see people caring is the large variety of {{IOException}} s.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
> IOExceptions should contain the filename of the broken input files
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1921
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1921
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Krishna Ramachandran
> Assignee: Krishna Ramachandran
> Attachments: mapred-1921-1.patch, mapred-1921-3.patch, mapreduce-1921.patch
>
>
> If bzip or other decompression fails, the IOException does not contain the name of the broken file that caused the exception.
> It would be nice if such actions could be avoided in the future by having the name of the files that are broken spelled
> out in the exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1921) IOExceptions should contain the
filename of the broken input files
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Krishna Ramachandran updated MAPREDUCE-1921:
--------------------------------------------
Status: Open (was: Patch Available)
> IOExceptions should contain the filename of the broken input files
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1921
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1921
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Krishna Ramachandran
> Assignee: Krishna Ramachandran
> Attachments: mapred-1921-1.patch, mapreduce-1921.patch
>
>
> If bzip or other decompression fails, the IOException does not contain the name of the broken file that caused the exception.
> It would be nice if such actions could be avoided in the future by having the name of the files that are broken spelled
> out in the exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1921) IOExceptions should contain the
filename of the broken input files
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885855#action_12885855 ]
Hadoop QA commented on MAPREDUCE-1921:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12448846/mapreduce-1921.patch
against trunk revision 960808.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 2 new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/289/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/289/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/289/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/289/console
This message is automatically generated.
> IOExceptions should contain the filename of the broken input files
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1921
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1921
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Krishna Ramachandran
> Assignee: Krishna Ramachandran
> Attachments: mapreduce-1921.patch
>
>
> If bzip or other decompression fails, the IOException does not contain the name of the broken file that caused the exception.
> It would be nice if such actions could be avoided in the future by having the name of the files that are broken spelled
> out in the exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1921) IOExceptions should contain the
filename of the broken input files
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Krishna Ramachandran updated MAPREDUCE-1921:
--------------------------------------------
Attachment: mapreduce-1921.patch
patch to include filename in i/o exception
> IOExceptions should contain the filename of the broken input files
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1921
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1921
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Krishna Ramachandran
> Assignee: Krishna Ramachandran
> Attachments: mapreduce-1921.patch
>
>
> If bzip or other decompression fails, the IOException does not contain the name of the broken file that caused the exception.
> It would be nice if such actions could be avoided in the future by having the name of the files that are broken spelled
> out in the exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1921) IOExceptions should contain the
filename of the broken input files
Posted by "Dick King (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895096#action_12895096 ]
Dick King commented on MAPREDUCE-1921:
--------------------------------------
{{MapTask.java}}
When creating a new exception to re-throw, perhaps we should duplicate the original exception's type? I realize that there's a lot of code that doesn't do that, but swallowing this information feels wrong. Especially in the case of IO exceptions, I can envision a lot of code that wants to treat certain sub-exceptions a bit differently; some IO exceptions are the caller's fault but some represent oddities that happened in the file system.
Perhaps something like
{noformat}
throw (IOException)(ioe.getClass().getConstructor(String.class, ioe.getClass())
.newInstance("IO error ..." + ... , ioe));
{noformat}
?
This cliche should occur in two places; perhaps we should pull it out and put it in one of the utilities classes? Perhaps IO utilities, since the main use case where I can see people caring is the large variety of {{IOException}} s.
> IOExceptions should contain the filename of the broken input files
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1921
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1921
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Krishna Ramachandran
> Assignee: Krishna Ramachandran
> Attachments: mapred-1921-1.patch, mapred-1921-3.patch, mapreduce-1921.patch
>
>
> If bzip or other decompression fails, the IOException does not contain the name of the broken file that caused the exception.
> It would be nice if such actions could be avoided in the future by having the name of the files that are broken spelled
> out in the exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1921) IOExceptions should contain the
filename of the broken input files
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Krishna Ramachandran updated MAPREDUCE-1921:
--------------------------------------------
Attachment: mapred-1921-3.patch
patch updated to include latest file changes in trunk
> IOExceptions should contain the filename of the broken input files
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1921
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1921
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Krishna Ramachandran
> Assignee: Krishna Ramachandran
> Attachments: mapred-1921-1.patch, mapred-1921-3.patch, mapreduce-1921.patch
>
>
> If bzip or other decompression fails, the IOException does not contain the name of the broken file that caused the exception.
> It would be nice if such actions could be avoided in the future by having the name of the files that are broken spelled
> out in the exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1921) IOExceptions should contain the
filename of the broken input files
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Krishna Ramachandran updated MAPREDUCE-1921:
--------------------------------------------
Attachment: mapred-1921-1.patch
revised - fix findbug warning
> IOExceptions should contain the filename of the broken input files
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1921
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1921
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Krishna Ramachandran
> Assignee: Krishna Ramachandran
> Attachments: mapred-1921-1.patch, mapreduce-1921.patch
>
>
> If bzip or other decompression fails, the IOException does not contain the name of the broken file that caused the exception.
> It would be nice if such actions could be avoided in the future by having the name of the files that are broken spelled
> out in the exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1921) IOExceptions should contain the
filename of the broken input files
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Krishna Ramachandran updated MAPREDUCE-1921:
--------------------------------------------
Status: Patch Available (was: Open)
> IOExceptions should contain the filename of the broken input files
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1921
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1921
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Krishna Ramachandran
> Assignee: Krishna Ramachandran
> Attachments: mapred-1921-1.patch, mapreduce-1921.patch
>
>
> If bzip or other decompression fails, the IOException does not contain the name of the broken file that caused the exception.
> It would be nice if such actions could be avoided in the future by having the name of the files that are broken spelled
> out in the exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1921) IOExceptions should contain the
filename of the broken input files
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886356#action_12886356 ]
Krishna Ramachandran commented on MAPREDUCE-1921:
-------------------------------------------------
Yes I will fix the error handling
Test case - the exception was swallowed somewhere in jobclient - but file name is seen in test logs
Also manual testing with a broken bz2 correctly displayed the file name
Let me fix the test (or with a real a broken bz2 copied to dfs) and revise
> IOExceptions should contain the filename of the broken input files
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1921
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1921
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Krishna Ramachandran
> Assignee: Krishna Ramachandran
> Attachments: mapred-1921-1.patch, mapreduce-1921.patch
>
>
> If bzip or other decompression fails, the IOException does not contain the name of the broken file that caused the exception.
> It would be nice if such actions could be avoided in the future by having the name of the files that are broken spelled
> out in the exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1921) IOExceptions should contain the
filename of the broken input files
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Krishna Ramachandran updated MAPREDUCE-1921:
--------------------------------------------
Status: Patch Available (was: Open)
> IOExceptions should contain the filename of the broken input files
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1921
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1921
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Krishna Ramachandran
> Assignee: Krishna Ramachandran
> Attachments: mapreduce-1921.patch
>
>
> If bzip or other decompression fails, the IOException does not contain the name of the broken file that caused the exception.
> It would be nice if such actions could be avoided in the future by having the name of the files that are broken spelled
> out in the exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1921) IOExceptions should contain the
filename of the broken input files
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-1921:
-------------------------------------
Status: Open (was: Patch Available)
Krishna, some comments:
1. Minor nit: The exception handling code could be slightly better in both moveToNext and nextKeyValue as:
protected synchronized boolean moveToNext(K key, V value)
throws IOException {
- reporter.setProgress(getProgress());
- beforePos = getPos();
- boolean ret = rawIn.next(key, value);
- afterPos = getPos();
+ boolean ret = false;
+ try {
+ reporter.setProgress(getProgress());
+ beforePos = getPos();
+ ret = rawIn.next(key, value);
+ afterPos = getPos();
+ } catch (IOException ioe) {
+ if (split instanceof FileSplit) {
+ LOG.error("IO error in map input file " + conf.get("map.input.file"));
+ throw new IOException("IO error in map input file "
+ + conf.get("map.input.file"), ioe);
+ }
+ }
return ret;
2. The test-case doesn't seem to check for the file name in the failure?
> IOExceptions should contain the filename of the broken input files
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1921
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1921
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Krishna Ramachandran
> Assignee: Krishna Ramachandran
> Attachments: mapred-1921-1.patch, mapreduce-1921.patch
>
>
> If bzip or other decompression fails, the IOException does not contain the name of the broken file that caused the exception.
> It would be nice if such actions could be avoided in the future by having the name of the files that are broken spelled
> out in the exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1921) IOExceptions should contain the
filename of the broken input files
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886029#action_12886029 ]
Hadoop QA commented on MAPREDUCE-1921:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12448874/mapred-1921-1.patch
against trunk revision 960808.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/291/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/291/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/291/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/291/console
This message is automatically generated.
> IOExceptions should contain the filename of the broken input files
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-1921
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1921
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Krishna Ramachandran
> Assignee: Krishna Ramachandran
> Attachments: mapred-1921-1.patch, mapreduce-1921.patch
>
>
> If bzip or other decompression fails, the IOException does not contain the name of the broken file that caused the exception.
> It would be nice if such actions could be avoided in the future by having the name of the files that are broken spelled
> out in the exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.