You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Wouter de Bie (Created) (JIRA)" <ji...@apache.org> on 2012/04/11 14:02:17 UTC
[jira] [Created] (MAPREDUCE-4136) Hadoop streaming might succeed
even through reducer fails
Hadoop streaming might succeed even through reducer fails
---------------------------------------------------------
Key: MAPREDUCE-4136
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4136
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming
Affects Versions: 0.20.205.0
Reporter: Wouter de Bie
Hadoop streaming can even succeed even though the reducer has failed. This happens when Hadoop calls {{PipeReducer.close}}, but in the mean time the reducer has failed and the process has died. When {{clientOut_.flush()}} throws an {{IOException}} in {{PipeMapRed.mapRedFinish()}} this exception is caught but only logged. The exit status of the child process is never checked and task is marked as successful.
We're working on a patch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4136) Hadoop streaming might succeed
even through reducer fails
Posted by "Robert Joseph Evans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13251853#comment-13251853 ]
Robert Joseph Evans commented on MAPREDUCE-4136:
------------------------------------------------
A similar fix was already done for MAPREDUCE-3790. I don't really know if this is the same fix or not, but the patch only went into trunk, and branch-0.23/branch-2. So it might be worth looking at to see if there is some overlap between the two.
> Hadoop streaming might succeed even through reducer fails
> ---------------------------------------------------------
>
> Key: MAPREDUCE-4136
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4136
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/streaming
> Affects Versions: 0.20.205.0
> Reporter: Wouter de Bie
> Attachments: mapreduce-4136.patch
>
>
> Hadoop streaming can even succeed even though the reducer has failed. This happens when Hadoop calls {{PipeReducer.close()}}, but in the mean time the reducer has failed and the process has died. When {{clientOut_.flush()}} throws an {{IOException}} in {{PipeMapRed.mapRedFinish()}} this exception is caught but only logged. The exit status of the child process is never checked and task is marked as successful.
> I've attached a patch that seems to fix it for us.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4136) Hadoop streaming might succeed
even through reducer fails
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13251501#comment-13251501 ]
Hadoop QA commented on MAPREDUCE-4136:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12522230/mapreduce-4136.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2196//console
This message is automatically generated.
> Hadoop streaming might succeed even through reducer fails
> ---------------------------------------------------------
>
> Key: MAPREDUCE-4136
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4136
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/streaming
> Affects Versions: 0.20.205.0
> Reporter: Wouter de Bie
> Attachments: mapreduce-4136.patch
>
>
> Hadoop streaming can even succeed even though the reducer has failed. This happens when Hadoop calls {{PipeReducer.close()}}, but in the mean time the reducer has failed and the process has died. When {{clientOut_.flush()}} throws an {{IOException}} in {{PipeMapRed.mapRedFinish()}} this exception is caught but only logged. The exit status of the child process is never checked and task is marked as successful.
> I've attached a patch that seems to fix it for us.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4136) Hadoop streaming might succeed
even through reducer fails
Posted by "Wouter de Bie (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wouter de Bie updated MAPREDUCE-4136:
-------------------------------------
Description:
Hadoop streaming can even succeed even though the reducer has failed. This happens when Hadoop calls {{PipeReducer.close()}}, but in the mean time the reducer has failed and the process has died. When {{clientOut_.flush()}} throws an {{IOException}} in {{PipeMapRed.mapRedFinish()}} this exception is caught but only logged. The exit status of the child process is never checked and task is marked as successful.
We're working on a patch.
was:
Hadoop streaming can even succeed even though the reducer has failed. This happens when Hadoop calls {{PipeReducer.close}}, but in the mean time the reducer has failed and the process has died. When {{clientOut_.flush()}} throws an {{IOException}} in {{PipeMapRed.mapRedFinish()}} this exception is caught but only logged. The exit status of the child process is never checked and task is marked as successful.
We're working on a patch.
> Hadoop streaming might succeed even through reducer fails
> ---------------------------------------------------------
>
> Key: MAPREDUCE-4136
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4136
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/streaming
> Affects Versions: 0.20.205.0
> Reporter: Wouter de Bie
>
> Hadoop streaming can even succeed even though the reducer has failed. This happens when Hadoop calls {{PipeReducer.close()}}, but in the mean time the reducer has failed and the process has died. When {{clientOut_.flush()}} throws an {{IOException}} in {{PipeMapRed.mapRedFinish()}} this exception is caught but only logged. The exit status of the child process is never checked and task is marked as successful.
> We're working on a patch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4136) Hadoop streaming might succeed
even through reducer fails
Posted by "Wouter de Bie (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wouter de Bie updated MAPREDUCE-4136:
-------------------------------------
Attachment: mapreduce-4136.patch
> Hadoop streaming might succeed even through reducer fails
> ---------------------------------------------------------
>
> Key: MAPREDUCE-4136
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4136
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/streaming
> Affects Versions: 0.20.205.0
> Reporter: Wouter de Bie
> Attachments: mapreduce-4136.patch
>
>
> Hadoop streaming can even succeed even though the reducer has failed. This happens when Hadoop calls {{PipeReducer.close()}}, but in the mean time the reducer has failed and the process has died. When {{clientOut_.flush()}} throws an {{IOException}} in {{PipeMapRed.mapRedFinish()}} this exception is caught but only logged. The exit status of the child process is never checked and task is marked as successful.
> We're working on a patch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4136) Hadoop streaming might succeed
even through reducer fails
Posted by "Wouter de Bie (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wouter de Bie updated MAPREDUCE-4136:
-------------------------------------
Description:
Hadoop streaming can even succeed even though the reducer has failed. This happens when Hadoop calls {{PipeReducer.close()}}, but in the mean time the reducer has failed and the process has died. When {{clientOut_.flush()}} throws an {{IOException}} in {{PipeMapRed.mapRedFinish()}} this exception is caught but only logged. The exit status of the child process is never checked and task is marked as successful.
I've attached a patch that seems to fix it for us.
was:
Hadoop streaming can even succeed even though the reducer has failed. This happens when Hadoop calls {{PipeReducer.close()}}, but in the mean time the reducer has failed and the process has died. When {{clientOut_.flush()}} throws an {{IOException}} in {{PipeMapRed.mapRedFinish()}} this exception is caught but only logged. The exit status of the child process is never checked and task is marked as successful.
We're working on a patch.
> Hadoop streaming might succeed even through reducer fails
> ---------------------------------------------------------
>
> Key: MAPREDUCE-4136
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4136
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/streaming
> Affects Versions: 0.20.205.0
> Reporter: Wouter de Bie
> Attachments: mapreduce-4136.patch
>
>
> Hadoop streaming can even succeed even though the reducer has failed. This happens when Hadoop calls {{PipeReducer.close()}}, but in the mean time the reducer has failed and the process has died. When {{clientOut_.flush()}} throws an {{IOException}} in {{PipeMapRed.mapRedFinish()}} this exception is caught but only logged. The exit status of the child process is never checked and task is marked as successful.
> I've attached a patch that seems to fix it for us.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4136) Hadoop streaming might succeed
even through reducer fails
Posted by "Matteo Bertozzi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291260#comment-13291260 ]
Matteo Bertozzi commented on MAPREDUCE-4136:
--------------------------------------------
Yes this is fixed with MAPREDUCE-3790, the IOException during clientOut._flush() is now catched.
> Hadoop streaming might succeed even through reducer fails
> ---------------------------------------------------------
>
> Key: MAPREDUCE-4136
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4136
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/streaming
> Affects Versions: 0.20.205.0
> Reporter: Wouter de Bie
> Attachments: mapreduce-4136.patch
>
>
> Hadoop streaming can even succeed even though the reducer has failed. This happens when Hadoop calls {{PipeReducer.close()}}, but in the mean time the reducer has failed and the process has died. When {{clientOut_.flush()}} throws an {{IOException}} in {{PipeMapRed.mapRedFinish()}} this exception is caught but only logged. The exit status of the child process is never checked and task is marked as successful.
> I've attached a patch that seems to fix it for us.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4136) Hadoop streaming might succeed
even through reducer fails
Posted by "Wouter de Bie (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wouter de Bie updated MAPREDUCE-4136:
-------------------------------------
Status: Patch Available (was: Open)
> Hadoop streaming might succeed even through reducer fails
> ---------------------------------------------------------
>
> Key: MAPREDUCE-4136
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4136
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/streaming
> Affects Versions: 0.20.205.0
> Reporter: Wouter de Bie
> Attachments: mapreduce-4136.patch
>
>
> Hadoop streaming can even succeed even though the reducer has failed. This happens when Hadoop calls {{PipeReducer.close()}}, but in the mean time the reducer has failed and the process has died. When {{clientOut_.flush()}} throws an {{IOException}} in {{PipeMapRed.mapRedFinish()}} this exception is caught but only logged. The exit status of the child process is never checked and task is marked as successful.
> We're working on a patch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira