You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Ning Zhang (JIRA)" <ji...@apache.org> on 2010/08/16 22:09:17 UTC
[jira] Created: (HIVE-1543) set abort in ExecMapper when Hive's
record reader got an IOException
set abort in ExecMapper when Hive's record reader got an IOException
--------------------------------------------------------------------
Key: HIVE-1543
URL: https://issues.apache.org/jira/browse/HIVE-1543
Project: Hadoop Hive
Issue Type: Improvement
Reporter: Ning Zhang
Assignee: Ning Zhang
When RecordReader got an IOException, ExecMapper does not know and will close the operators as if there is not error. We should catch this exception and avoid writing partial results to HDFS which will be removed later anyways.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1543) set abort in ExecMapper when Hive's
record reader got an IOException
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899207#action_12899207 ]
Namit Jain commented on HIVE-1543:
----------------------------------
can you reload the patch for 0.6 - i got some merge conflicts
> set abort in ExecMapper when Hive's record reader got an IOException
> --------------------------------------------------------------------
>
> Key: HIVE-1543
> URL: https://issues.apache.org/jira/browse/HIVE-1543
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Fix For: 0.6.0
>
> Attachments: HIVE-1543.1.patch, HIVE-1543.patch, HIVE-1543_branch0.6.patch
>
>
> When RecordReader got an IOException, ExecMapper does not know and will close the operators as if there is not error. We should catch this exception and avoid writing partial results to HDFS which will be removed later anyways.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1543) set abort in ExecMapper when Hive's
record reader got an IOException
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899209#action_12899209 ]
Namit Jain commented on HIVE-1543:
----------------------------------
+1 for trunk
> set abort in ExecMapper when Hive's record reader got an IOException
> --------------------------------------------------------------------
>
> Key: HIVE-1543
> URL: https://issues.apache.org/jira/browse/HIVE-1543
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Fix For: 0.6.0
>
> Attachments: HIVE-1543.1.patch, HIVE-1543.patch, HIVE-1543_branch0.6.patch
>
>
> When RecordReader got an IOException, ExecMapper does not know and will close the operators as if there is not error. We should catch this exception and avoid writing partial results to HDFS which will be removed later anyways.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1543) set abort in ExecMapper when Hive's
record reader got an IOException
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-1543:
---------------------------------
Component/s: Query Processor
> set abort in ExecMapper when Hive's record reader got an IOException
> --------------------------------------------------------------------
>
> Key: HIVE-1543
> URL: https://issues.apache.org/jira/browse/HIVE-1543
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Fix For: 0.6.0
>
> Attachments: HIVE-1543.1.patch, HIVE-1543.2_branch0.6.patch, HIVE-1543.patch, HIVE-1543_branch0.6.patch
>
>
> When RecordReader got an IOException, ExecMapper does not know and will close the operators as if there is not error. We should catch this exception and avoid writing partial results to HDFS which will be removed later anyways.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1543) set abort in ExecMapper when Hive's
record reader got an IOException
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang updated HIVE-1543:
-----------------------------
Attachment: HIVE-1543.patch
passed all hadoop 0.20 tests.
> set abort in ExecMapper when Hive's record reader got an IOException
> --------------------------------------------------------------------
>
> Key: HIVE-1543
> URL: https://issues.apache.org/jira/browse/HIVE-1543
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Attachments: HIVE-1543.patch
>
>
> When RecordReader got an IOException, ExecMapper does not know and will close the operators as if there is not error. We should catch this exception and avoid writing partial results to HDFS which will be removed later anyways.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1543) set abort in ExecMapper when Hive's
record reader got an IOException
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899098#action_12899098 ]
He Yongqiang commented on HIVE-1543:
------------------------------------
let's do it in HiveContextAwareRecordReader. And maybe store the var in IOContext?
> set abort in ExecMapper when Hive's record reader got an IOException
> --------------------------------------------------------------------
>
> Key: HIVE-1543
> URL: https://issues.apache.org/jira/browse/HIVE-1543
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Fix For: 0.7.0
>
> Attachments: HIVE-1543.patch
>
>
> When RecordReader got an IOException, ExecMapper does not know and will close the operators as if there is not error. We should catch this exception and avoid writing partial results to HDFS which will be removed later anyways.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1543) set abort in ExecMapper when Hive's
record reader got an IOException
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang updated HIVE-1543:
-----------------------------
Fix Version/s: 0.6.0
(was: 0.7.0)
> set abort in ExecMapper when Hive's record reader got an IOException
> --------------------------------------------------------------------
>
> Key: HIVE-1543
> URL: https://issues.apache.org/jira/browse/HIVE-1543
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Fix For: 0.6.0
>
> Attachments: HIVE-1543.1.patch, HIVE-1543.patch, HIVE-1543_branch0.6.patch
>
>
> When RecordReader got an IOException, ExecMapper does not know and will close the operators as if there is not error. We should catch this exception and avoid writing partial results to HDFS which will be removed later anyways.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1543) set abort in ExecMapper when Hive's
record reader got an IOException
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang updated HIVE-1543:
-----------------------------
Attachment: HIVE-1543.1.patch
HIVE-1543_branch0.6.patch
Uploading 2 patches for trunk and 0.6 respectively.
> set abort in ExecMapper when Hive's record reader got an IOException
> --------------------------------------------------------------------
>
> Key: HIVE-1543
> URL: https://issues.apache.org/jira/browse/HIVE-1543
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Fix For: 0.6.0
>
> Attachments: HIVE-1543.1.patch, HIVE-1543.patch, HIVE-1543_branch0.6.patch
>
>
> When RecordReader got an IOException, ExecMapper does not know and will close the operators as if there is not error. We should catch this exception and avoid writing partial results to HDFS which will be removed later anyways.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1543) set abort in ExecMapper when Hive's
record reader got an IOException
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899169#action_12899169 ]
He Yongqiang commented on HIVE-1543:
------------------------------------
we can do two different patches for trunk and 0.6.
I think BucketizedHiveRecordReader also extends HiveContextAwareRecordReader.
> set abort in ExecMapper when Hive's record reader got an IOException
> --------------------------------------------------------------------
>
> Key: HIVE-1543
> URL: https://issues.apache.org/jira/browse/HIVE-1543
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Fix For: 0.7.0
>
> Attachments: HIVE-1543.patch
>
>
> When RecordReader got an IOException, ExecMapper does not know and will close the operators as if there is not error. We should catch this exception and avoid writing partial results to HDFS which will be removed later anyways.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1543) set abort in ExecMapper when Hive's
record reader got an IOException
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899158#action_12899158 ]
Ning Zhang commented on HIVE-1543:
----------------------------------
@yongqiang, HiveContextAwareRecordReader is only available in 0.7 (trunk) but not 0.6. I think this should be back ported to 0.6 as well (as we did for HIVE-1492)?
> set abort in ExecMapper when Hive's record reader got an IOException
> --------------------------------------------------------------------
>
> Key: HIVE-1543
> URL: https://issues.apache.org/jira/browse/HIVE-1543
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Fix For: 0.7.0
>
> Attachments: HIVE-1543.patch
>
>
> When RecordReader got an IOException, ExecMapper does not know and will close the operators as if there is not error. We should catch this exception and avoid writing partial results to HDFS which will be removed later anyways.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1543) set abort in ExecMapper when Hive's
record reader got an IOException
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899165#action_12899165 ]
Namit Jain commented on HIVE-1543:
----------------------------------
What about BucketizedHiveRecordReader ?
> set abort in ExecMapper when Hive's record reader got an IOException
> --------------------------------------------------------------------
>
> Key: HIVE-1543
> URL: https://issues.apache.org/jira/browse/HIVE-1543
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Fix For: 0.7.0
>
> Attachments: HIVE-1543.patch
>
>
> When RecordReader got an IOException, ExecMapper does not know and will close the operators as if there is not error. We should catch this exception and avoid writing partial results to HDFS which will be removed later anyways.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1543) set abort in ExecMapper when Hive's
record reader got an IOException
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899225#action_12899225 ]
Namit Jain commented on HIVE-1543:
----------------------------------
+1 for 0.6 also
Will commit in both once the tests pass
> set abort in ExecMapper when Hive's record reader got an IOException
> --------------------------------------------------------------------
>
> Key: HIVE-1543
> URL: https://issues.apache.org/jira/browse/HIVE-1543
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Fix For: 0.6.0
>
> Attachments: HIVE-1543.1.patch, HIVE-1543.2_branch0.6.patch, HIVE-1543.patch, HIVE-1543_branch0.6.patch
>
>
> When RecordReader got an IOException, ExecMapper does not know and will close the operators as if there is not error. We should catch this exception and avoid writing partial results to HDFS which will be removed later anyways.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1543) set abort in ExecMapper when Hive's
record reader got an IOException
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang updated HIVE-1543:
-----------------------------
Status: Patch Available (was: Open)
Fix Version/s: 0.7.0
> set abort in ExecMapper when Hive's record reader got an IOException
> --------------------------------------------------------------------
>
> Key: HIVE-1543
> URL: https://issues.apache.org/jira/browse/HIVE-1543
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Fix For: 0.7.0
>
> Attachments: HIVE-1543.patch
>
>
> When RecordReader got an IOException, ExecMapper does not know and will close the operators as if there is not error. We should catch this exception and avoid writing partial results to HDFS which will be removed later anyways.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1543) set abort in ExecMapper when Hive's
record reader got an IOException
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang updated HIVE-1543:
-----------------------------
Attachment: HIVE-1543.2_branch0.6.patch
Uploading a new patch for branch 0.6.
> set abort in ExecMapper when Hive's record reader got an IOException
> --------------------------------------------------------------------
>
> Key: HIVE-1543
> URL: https://issues.apache.org/jira/browse/HIVE-1543
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Fix For: 0.6.0
>
> Attachments: HIVE-1543.1.patch, HIVE-1543.2_branch0.6.patch, HIVE-1543.patch, HIVE-1543_branch0.6.patch
>
>
> When RecordReader got an IOException, ExecMapper does not know and will close the operators as if there is not error. We should catch this exception and avoid writing partial results to HDFS which will be removed later anyways.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1543) set abort in ExecMapper when Hive's
record reader got an IOException
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-1543:
-----------------------------
Status: Resolved (was: Patch Available)
Hadoop Flags: [Reviewed]
Resolution: Fixed
Committed. Thanks Ning
> set abort in ExecMapper when Hive's record reader got an IOException
> --------------------------------------------------------------------
>
> Key: HIVE-1543
> URL: https://issues.apache.org/jira/browse/HIVE-1543
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Fix For: 0.6.0
>
> Attachments: HIVE-1543.1.patch, HIVE-1543.2_branch0.6.patch, HIVE-1543.patch, HIVE-1543_branch0.6.patch
>
>
> When RecordReader got an IOException, ExecMapper does not know and will close the operators as if there is not error. We should catch this exception and avoid writing partial results to HDFS which will be removed later anyways.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.