You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Daniel Dai (JIRA)" <ji...@apache.org> on 2012/10/09 09:46:04 UTC
[jira] [Created] (HADOOP-8904) Hadoop does not close output file /
does not call Mapper.cleanup if exception in map
Daniel Dai created HADOOP-8904:
----------------------------------
Summary: Hadoop does not close output file / does not call Mapper.cleanup if exception in map
Key: HADOOP-8904
URL: https://issues.apache.org/jira/browse/HADOOP-8904
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 1-win
Reporter: Daniel Dai
Find this in Pig unit test TestStore under Windows. There are dangling files because map does not close the file when exception happens in map(). In Windows, Hadoop will not remove a file if it is not closed. This happens in reduce() as well.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8904) Hadoop does not close output file /
does not call Mapper.cleanup if exception in map
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated HADOOP-8904:
-------------------------------
Status: Patch Available (was: Open)
> Hadoop does not close output file / does not call Mapper.cleanup if exception in map
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8904
> URL: https://issues.apache.org/jira/browse/HADOOP-8904
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Daniel Dai
> Attachments: HADOOP-8904-1.patch
>
>
> Find this in Pig unit test TestStore under Windows. There are dangling files because map does not close the file when exception happens in map(). In Windows, Hadoop will not remove a file if it is not closed. This happens in reduce() as well.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8904) Hadoop does not close output file
/ does not call Mapper.cleanup if exception in map
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13479073#comment-13479073 ]
Tom White commented on HADOOP-8904:
-----------------------------------
This change makes the behaviour the same as the old API where the Mapper's close() method is called in a finally block - which is a good thing. Will it cause incompatibilities with existing code - i.e. any that assume cleanup() won't be called if the map throws an exception?
We should at least mark this as an incompatible change with a note saying that you need to override the Mapper's (or Reducer's) run() method to restore the old behaviour.
> Hadoop does not close output file / does not call Mapper.cleanup if exception in map
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8904
> URL: https://issues.apache.org/jira/browse/HADOOP-8904
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Attachments: HADOOP-23-2.patch, HADOOP-8904-1.patch
>
>
> Find this in Pig unit test TestStore under Windows. There are dangling files because map does not close the file when exception happens in map(). In Windows, Hadoop will not remove a file if it is not closed. This happens in reduce() as well.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8904) Hadoop does not close output file
/ does not call Mapper.cleanup if exception in map
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478760#comment-13478760 ]
Hadoop QA commented on HADOOP-8904:
-----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12549646/HADOOP-23-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1644//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1644//console
This message is automatically generated.
> Hadoop does not close output file / does not call Mapper.cleanup if exception in map
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8904
> URL: https://issues.apache.org/jira/browse/HADOOP-8904
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Attachments: HADOOP-23-2.patch, HADOOP-8904-1.patch
>
>
> Find this in Pig unit test TestStore under Windows. There are dangling files because map does not close the file when exception happens in map(). In Windows, Hadoop will not remove a file if it is not closed. This happens in reduce() as well.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8904) Hadoop does not close output file
/ does not call Mapper.cleanup if exception in map
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478648#comment-13478648 ]
Suresh Srinivas commented on HADOOP-8904:
-----------------------------------------
Daniel, can you please also change the indentation in this patch to two spaces.
> Hadoop does not close output file / does not call Mapper.cleanup if exception in map
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8904
> URL: https://issues.apache.org/jira/browse/HADOOP-8904
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Daniel Dai
> Attachments: HADOOP-8904-1.patch
>
>
> Find this in Pig unit test TestStore under Windows. There are dangling files because map does not close the file when exception happens in map(). In Windows, Hadoop will not remove a file if it is not closed. This happens in reduce() as well.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8904) Hadoop does not close output file
/ does not call Mapper.cleanup if exception in map
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472207#comment-13472207 ]
Hadoop QA commented on HADOOP-8904:
-----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12548375/HADOOP-8904-1.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1588//console
This message is automatically generated.
> Hadoop does not close output file / does not call Mapper.cleanup if exception in map
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8904
> URL: https://issues.apache.org/jira/browse/HADOOP-8904
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Daniel Dai
> Attachments: HADOOP-8904-1.patch
>
>
> Find this in Pig unit test TestStore under Windows. There are dangling files because map does not close the file when exception happens in map(). In Windows, Hadoop will not remove a file if it is not closed. This happens in reduce() as well.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8904) Hadoop does not close output file
/ does not call Mapper.cleanup if exception in map
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472549#comment-13472549 ]
Todd Lipcon commented on HADOOP-8904:
-------------------------------------
This seems like a bug fix which is relevant for branch-1 as well as newer branches, not just branch-1-win, right? Is trunk/branch-2 affected?
> Hadoop does not close output file / does not call Mapper.cleanup if exception in map
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8904
> URL: https://issues.apache.org/jira/browse/HADOOP-8904
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Daniel Dai
> Attachments: HADOOP-8904-1.patch
>
>
> Find this in Pig unit test TestStore under Windows. There are dangling files because map does not close the file when exception happens in map(). In Windows, Hadoop will not remove a file if it is not closed. This happens in reduce() as well.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8904) Hadoop does not close output file /
does not call Mapper.cleanup if exception in map
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated HADOOP-8904:
-------------------------------
Attachment: HADOOP-8904-1.patch
> Hadoop does not close output file / does not call Mapper.cleanup if exception in map
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8904
> URL: https://issues.apache.org/jira/browse/HADOOP-8904
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Daniel Dai
> Attachments: HADOOP-8904-1.patch
>
>
> Find this in Pig unit test TestStore under Windows. There are dangling files because map does not close the file when exception happens in map(). In Windows, Hadoop will not remove a file if it is not closed. This happens in reduce() as well.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8904) Hadoop does not close output file
/ does not call Mapper.cleanup if exception in map
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478721#comment-13478721 ]
Daniel Dai commented on HADOOP-8904:
------------------------------------
[~tlipcon] yes, it is for all branches, not just branch-1-win. The patch itself is for trunk, may need rebase to other branches
[~sureshms] I updated the patch for the format issues
> Hadoop does not close output file / does not call Mapper.cleanup if exception in map
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8904
> URL: https://issues.apache.org/jira/browse/HADOOP-8904
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Attachments: HADOOP-23-2.patch, HADOOP-8904-1.patch
>
>
> Find this in Pig unit test TestStore under Windows. There are dangling files because map does not close the file when exception happens in map(). In Windows, Hadoop will not remove a file if it is not closed. This happens in reduce() as well.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-8904) Hadoop does not close output file
/ does not call Mapper.cleanup if exception in map
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas reassigned HADOOP-8904:
---------------------------------------
Assignee: Daniel Dai
> Hadoop does not close output file / does not call Mapper.cleanup if exception in map
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8904
> URL: https://issues.apache.org/jira/browse/HADOOP-8904
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Attachments: HADOOP-8904-1.patch
>
>
> Find this in Pig unit test TestStore under Windows. There are dangling files because map does not close the file when exception happens in map(). In Windows, Hadoop will not remove a file if it is not closed. This happens in reduce() as well.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8904) Hadoop does not close output file /
does not call Mapper.cleanup if exception in map
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated HADOOP-8904:
-------------------------------
Attachment: HADOOP-23-2.patch
> Hadoop does not close output file / does not call Mapper.cleanup if exception in map
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8904
> URL: https://issues.apache.org/jira/browse/HADOOP-8904
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Attachments: HADOOP-23-2.patch, HADOOP-8904-1.patch
>
>
> Find this in Pig unit test TestStore under Windows. There are dangling files because map does not close the file when exception happens in map(). In Windows, Hadoop will not remove a file if it is not closed. This happens in reduce() as well.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira