You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Zheng Shao (JIRA)" <ji...@apache.org> on 2009/06/11 20:09:07 UTC
[jira] Created: (HIVE-557) Exception in FileSinkOperator's close
should NOT be ignored
Exception in FileSinkOperator's close should NOT be ignored
-----------------------------------------------------------
Key: HIVE-557
URL: https://issues.apache.org/jira/browse/HIVE-557
Project: Hadoop Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.4.0
Reporter: Zheng Shao
Assignee: Zheng Shao
FileSinkOperator currently ignores all IOExceptions from close() and commit(). We should not ignore them, or the output file can be incomplete or missing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-557) Exception in FileSinkOperator's close
should NOT be ignored
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HIVE-557:
----------------------------
Attachment: HIVE-557.2.patch
Changed Exception/HiveException to Throwable to places where we are setting "abort = true".
> Exception in FileSinkOperator's close should NOT be ignored
> -----------------------------------------------------------
>
> Key: HIVE-557
> URL: https://issues.apache.org/jira/browse/HIVE-557
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.3.0, 0.3.1
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Fix For: 0.4.0
>
> Attachments: HIVE-557.1.patch, HIVE-557.2.patch, HIVE-557.2.patch
>
>
> FileSinkOperator currently ignores all IOExceptions from close() and commit(). We should not ignore them, or the output file can be incomplete or missing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-557) Exception in FileSinkOperator's close
should NOT be ignored
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma updated HIVE-557:
-----------------------------------
Resolution: Fixed
Fix Version/s: 0.4.0
Status: Resolved (was: Patch Available)
committed. thanks Zheng!
> Exception in FileSinkOperator's close should NOT be ignored
> -----------------------------------------------------------
>
> Key: HIVE-557
> URL: https://issues.apache.org/jira/browse/HIVE-557
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.3.0, 0.3.1
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Fix For: 0.4.0
>
> Attachments: HIVE-557.1.patch, HIVE-557.2.patch
>
>
> FileSinkOperator currently ignores all IOExceptions from close() and commit(). We should not ignore them, or the output file can be incomplete or missing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-557) Exception in FileSinkOperator's close
should NOT be ignored
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HIVE-557:
----------------------------
Status: Patch Available (was: Open)
> Exception in FileSinkOperator's close should NOT be ignored
> -----------------------------------------------------------
>
> Key: HIVE-557
> URL: https://issues.apache.org/jira/browse/HIVE-557
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.4.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: HIVE-557.1.patch
>
>
> FileSinkOperator currently ignores all IOExceptions from close() and commit(). We should not ignore them, or the output file can be incomplete or missing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-557) Exception in FileSinkOperator's close
should NOT be ignored
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HIVE-557:
----------------------------
Attachment: HIVE-557.2.patch
Modified ExecMapper and ExecReducer to catch all Exceptions and set "abort".
> Exception in FileSinkOperator's close should NOT be ignored
> -----------------------------------------------------------
>
> Key: HIVE-557
> URL: https://issues.apache.org/jira/browse/HIVE-557
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.4.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: HIVE-557.1.patch, HIVE-557.2.patch
>
>
> FileSinkOperator currently ignores all IOExceptions from close() and commit(). We should not ignore them, or the output file can be incomplete or missing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-557) Exception in FileSinkOperator's close
should NOT be ignored
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718638#action_12718638 ]
Joydeep Sen Sarma commented on HIVE-557:
----------------------------------------
just talked to Zheng - need to catch all Exceptions from mapoperator and not just hiveexception ..
> Exception in FileSinkOperator's close should NOT be ignored
> -----------------------------------------------------------
>
> Key: HIVE-557
> URL: https://issues.apache.org/jira/browse/HIVE-557
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.4.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: HIVE-557.1.patch
>
>
> FileSinkOperator currently ignores all IOExceptions from close() and commit(). We should not ignore them, or the output file can be incomplete or missing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-557) Exception in FileSinkOperator's close
should NOT be ignored
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma updated HIVE-557:
-----------------------------------
Affects Version/s: (was: 0.4.0)
0.3.1
0.3.0
> Exception in FileSinkOperator's close should NOT be ignored
> -----------------------------------------------------------
>
> Key: HIVE-557
> URL: https://issues.apache.org/jira/browse/HIVE-557
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.3.0, 0.3.1
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Fix For: 0.4.0
>
> Attachments: HIVE-557.1.patch, HIVE-557.2.patch
>
>
> FileSinkOperator currently ignores all IOExceptions from close() and commit(). We should not ignore them, or the output file can be incomplete or missing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (HIVE-557) Exception in FileSinkOperator's close
should NOT be ignored
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao reopened HIVE-557:
-----------------------------
The "close()" is called in the "finally" block in "MapRunner.run()". If there is an out-of-memory error, Hive won't be able to catch it (because we only catch "Exception" now), so abort is NOT set to false, which is wrong.
We need to catch "Throwable" instead of "Exception".
> Exception in FileSinkOperator's close should NOT be ignored
> -----------------------------------------------------------
>
> Key: HIVE-557
> URL: https://issues.apache.org/jira/browse/HIVE-557
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.3.0, 0.3.1
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Fix For: 0.4.0
>
> Attachments: HIVE-557.1.patch, HIVE-557.2.patch
>
>
> FileSinkOperator currently ignores all IOExceptions from close() and commit(). We should not ignore them, or the output file can be incomplete or missing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-557) Exception in FileSinkOperator's close
should NOT be ignored
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HIVE-557:
----------------------------
Attachment: HIVE-557.1.patch
> Exception in FileSinkOperator's close should NOT be ignored
> -----------------------------------------------------------
>
> Key: HIVE-557
> URL: https://issues.apache.org/jira/browse/HIVE-557
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.4.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: HIVE-557.1.patch
>
>
> FileSinkOperator currently ignores all IOExceptions from close() and commit(). We should not ignore them, or the output file can be incomplete or missing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-557) Exception in FileSinkOperator's close
should NOT be ignored
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma resolved HIVE-557.
------------------------------------
Resolution: Fixed
commited again. thanks Zheng.
> Exception in FileSinkOperator's close should NOT be ignored
> -----------------------------------------------------------
>
> Key: HIVE-557
> URL: https://issues.apache.org/jira/browse/HIVE-557
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.3.0, 0.3.1
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Fix For: 0.4.0
>
> Attachments: HIVE-557.1.patch, HIVE-557.2.patch, HIVE-557.2.patch
>
>
> FileSinkOperator currently ignores all IOExceptions from close() and commit(). We should not ignore them, or the output file can be incomplete or missing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.