You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Ning Zhang (JIRA)" <ji...@apache.org> on 2011/02/09 20:33:57 UTC
[jira] Created: (HIVE-1980) Merging using mapreduce rather than
map-only job failed in case of dynamic partition inserts
Merging using mapreduce rather than map-only job failed in case of dynamic partition inserts
--------------------------------------------------------------------------------------------
Key: HIVE-1980
URL: https://issues.apache.org/jira/browse/HIVE-1980
Project: Hive
Issue Type: Bug
Reporter: Ning Zhang
Assignee: Ning Zhang
In dynamic partition insert and if merge is set to true and hive.mergejob.maponly=false, the merge MapReduce job will fail.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1980) Merging using mapreduce rather than
map-only job failed in case of dynamic partition inserts
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12997962#comment-12997962 ]
Namit Jain commented on HIVE-1980:
----------------------------------
Wont it also apply to non-dynamic partitions.
Won't it apply to all partitions where merge is being performed ?
> Merging using mapreduce rather than map-only job failed in case of dynamic partition inserts
> --------------------------------------------------------------------------------------------
>
> Key: HIVE-1980
> URL: https://issues.apache.org/jira/browse/HIVE-1980
> Project: Hive
> Issue Type: Bug
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Attachments: HIVE-1980.patch
>
>
> In dynamic partition insert and if merge is set to true and hive.mergejob.maponly=false, the merge MapReduce job will fail.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1980) Merging using mapreduce rather than
map-only job failed in case of dynamic partition inserts
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992753#comment-12992753 ]
Ning Zhang commented on HIVE-1980:
----------------------------------
I remembered that this case is not supported by design and we should throw an error in SemanticAnalyzer.
In the first dynamic partition insert patch we disabled merge completely because in the case of using HiveInputFormat (eg. in Hadoop 0.17) the partition columns are not passed to the reducer (part of the partition columns exist in the HDFS directory). So the reducer will create one file that may mix data from different partitions. In HIVE-1307 we enabled merge for CombineHiveInputFormat. However we should disable merge for the case of dynamic partition inserts using HiveInputFormat.
> Merging using mapreduce rather than map-only job failed in case of dynamic partition inserts
> --------------------------------------------------------------------------------------------
>
> Key: HIVE-1980
> URL: https://issues.apache.org/jira/browse/HIVE-1980
> Project: Hive
> Issue Type: Bug
> Reporter: Ning Zhang
> Assignee: Ning Zhang
>
> In dynamic partition insert and if merge is set to true and hive.mergejob.maponly=false, the merge MapReduce job will fail.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-1980) Merging using mapreduce rather than
map-only job failed in case of dynamic partition inserts
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-1980:
-----------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed. Thanks Ning
> Merging using mapreduce rather than map-only job failed in case of dynamic partition inserts
> --------------------------------------------------------------------------------------------
>
> Key: HIVE-1980
> URL: https://issues.apache.org/jira/browse/HIVE-1980
> Project: Hive
> Issue Type: Bug
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Attachments: HIVE-1980.patch
>
>
> In dynamic partition insert and if merge is set to true and hive.mergejob.maponly=false, the merge MapReduce job will fail.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-1980) Merging using mapreduce rather than
map-only job failed in case of dynamic partition inserts
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-1980:
---------------------------------
Component/s: Query Processor
Fix Version/s: 0.8.0
> Merging using mapreduce rather than map-only job failed in case of dynamic partition inserts
> --------------------------------------------------------------------------------------------
>
> Key: HIVE-1980
> URL: https://issues.apache.org/jira/browse/HIVE-1980
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Fix For: 0.8.0
>
> Attachments: HIVE-1980.patch
>
>
> In dynamic partition insert and if merge is set to true and hive.mergejob.maponly=false, the merge MapReduce job will fail.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-1980) Merging using mapreduce rather than
map-only job failed in case of dynamic partition inserts
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang updated HIVE-1980:
-----------------------------
Status: Patch Available (was: Open)
> Merging using mapreduce rather than map-only job failed in case of dynamic partition inserts
> --------------------------------------------------------------------------------------------
>
> Key: HIVE-1980
> URL: https://issues.apache.org/jira/browse/HIVE-1980
> Project: Hive
> Issue Type: Bug
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Attachments: HIVE-1980.patch
>
>
> In dynamic partition insert and if merge is set to true and hive.mergejob.maponly=false, the merge MapReduce job will fail.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-1980) Merging using mapreduce rather than
map-only job failed in case of dynamic partition inserts
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang updated HIVE-1980:
-----------------------------
Attachment: HIVE-1980.patch
> Merging using mapreduce rather than map-only job failed in case of dynamic partition inserts
> --------------------------------------------------------------------------------------------
>
> Key: HIVE-1980
> URL: https://issues.apache.org/jira/browse/HIVE-1980
> Project: Hive
> Issue Type: Bug
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Attachments: HIVE-1980.patch
>
>
> In dynamic partition insert and if merge is set to true and hive.mergejob.maponly=false, the merge MapReduce job will fail.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1980) Merging using mapreduce rather than
map-only job failed in case of dynamic partition inserts
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12997988#comment-12997988 ]
Namit Jain commented on HIVE-1980:
----------------------------------
Talked offline with Ning
+1
> Merging using mapreduce rather than map-only job failed in case of dynamic partition inserts
> --------------------------------------------------------------------------------------------
>
> Key: HIVE-1980
> URL: https://issues.apache.org/jira/browse/HIVE-1980
> Project: Hive
> Issue Type: Bug
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Attachments: HIVE-1980.patch
>
>
> In dynamic partition insert and if merge is set to true and hive.mergejob.maponly=false, the merge MapReduce job will fail.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1980) Merging using mapreduce rather than
map-only job failed in case of dynamic partition inserts
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12997976#comment-12997976 ]
Ning Zhang commented on HIVE-1980:
----------------------------------
This is not a problem for non-dynamic partition inserts because the input and output is a single directory/partition. So even if there is only 1 reducer the output will be a correct merge of the input files. In the dynamic partition case, we have to guarantee that a single reducer only takes input from 1 partition. Otherwise the result partition will be mixed with different input partitions.
> Merging using mapreduce rather than map-only job failed in case of dynamic partition inserts
> --------------------------------------------------------------------------------------------
>
> Key: HIVE-1980
> URL: https://issues.apache.org/jira/browse/HIVE-1980
> Project: Hive
> Issue Type: Bug
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Attachments: HIVE-1980.patch
>
>
> In dynamic partition insert and if merge is set to true and hive.mergejob.maponly=false, the merge MapReduce job will fail.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira