You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Namit Jain (JIRA)" <ji...@apache.org> on 2011/01/07 19:39:45 UTC
[jira] Created: (HIVE-1900) a mapper should be able to span
multiple partitions
a mapper should be able to span multiple partitions
---------------------------------------------------
Key: HIVE-1900
URL: https://issues.apache.org/jira/browse/HIVE-1900
Project: Hive
Issue Type: Improvement
Components: Query Processor
Reporter: Namit Jain
Assignee: He Yongqiang
Currently, a mapper only spans a single partition which creates a problem in the presence of many
small partitions (which is becoming a common usecase in facebook).
If the plan is the same, a mapper should be able to span files across multiple partitions
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1900) a mapper should be able to span
multiple partitions
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-1900:
-----------------------------
Attachment: hive.1900.1.patch
> a mapper should be able to span multiple partitions
> ---------------------------------------------------
>
> Key: HIVE-1900
> URL: https://issues.apache.org/jira/browse/HIVE-1900
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.1900.1.patch
>
>
> Currently, a mapper only spans a single partition which creates a problem in the presence of many
> small partitions (which is becoming a common usecase in facebook).
> If the plan is the same, a mapper should be able to span files across multiple partitions
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Assigned: (HIVE-1900) a mapper should be able to span
multiple partitions
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain reassigned HIVE-1900:
--------------------------------
Assignee: Namit Jain (was: Krishna Kumar)
> a mapper should be able to span multiple partitions
> ---------------------------------------------------
>
> Key: HIVE-1900
> URL: https://issues.apache.org/jira/browse/HIVE-1900
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.1900.1.patch, hive.1900.2.patch
>
>
> Currently, a mapper only spans a single partition which creates a problem in the presence of many
> small partitions (which is becoming a common usecase in facebook).
> If the plan is the same, a mapper should be able to span files across multiple partitions
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1900) a mapper should be able to span
multiple partitions
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978987#action_12978987 ]
Namit Jain commented on HIVE-1900:
----------------------------------
1. Why should it be any different than sort-merge join ?
ExecMapper needs to keep track of the current file, and then change the partitioning columns whenever the file changes
2. Why should it matter ?
3. Why should it matter ?
> a mapper should be able to span multiple partitions
> ---------------------------------------------------
>
> Key: HIVE-1900
> URL: https://issues.apache.org/jira/browse/HIVE-1900
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: He Yongqiang
>
> Currently, a mapper only spans a single partition which creates a problem in the presence of many
> small partitions (which is becoming a common usecase in facebook).
> If the plan is the same, a mapper should be able to span files across multiple partitions
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1900) a mapper should be able to span
multiple partitions
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990754#comment-12990754 ]
Namit Jain commented on HIVE-1900:
----------------------------------
New patch based on an offline review with Yongqiang
> a mapper should be able to span multiple partitions
> ---------------------------------------------------
>
> Key: HIVE-1900
> URL: https://issues.apache.org/jira/browse/HIVE-1900
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Krishna Kumar
> Attachments: hive.1900.1.patch, hive.1900.2.patch
>
>
> Currently, a mapper only spans a single partition which creates a problem in the presence of many
> small partitions (which is becoming a common usecase in facebook).
> If the plan is the same, a mapper should be able to span files across multiple partitions
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-1900) a mapper should be able to span
multiple partitions
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-1900:
-----------------------------
Status: Patch Available (was: Open)
I will create a review board request
> a mapper should be able to span multiple partitions
> ---------------------------------------------------
>
> Key: HIVE-1900
> URL: https://issues.apache.org/jira/browse/HIVE-1900
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.1900.1.patch
>
>
> Currently, a mapper only spans a single partition which creates a problem in the presence of many
> small partitions (which is becoming a common usecase in facebook).
> If the plan is the same, a mapper should be able to span files across multiple partitions
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Assigned: (HIVE-1900) a mapper should be able to span
multiple partitions
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain reassigned HIVE-1900:
--------------------------------
Assignee: Krishna Kumar (was: Namit Jain)
> a mapper should be able to span multiple partitions
> ---------------------------------------------------
>
> Key: HIVE-1900
> URL: https://issues.apache.org/jira/browse/HIVE-1900
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Krishna Kumar
> Attachments: hive.1900.1.patch
>
>
> Currently, a mapper only spans a single partition which creates a problem in the presence of many
> small partitions (which is becoming a common usecase in facebook).
> If the plan is the same, a mapper should be able to span files across multiple partitions
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-1900) a mapper should be able to span
multiple partitions
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
He Yongqiang updated HIVE-1900:
-------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed! Thanks Namit!
> a mapper should be able to span multiple partitions
> ---------------------------------------------------
>
> Key: HIVE-1900
> URL: https://issues.apache.org/jira/browse/HIVE-1900
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.1900.1.patch, hive.1900.2.patch
>
>
> Currently, a mapper only spans a single partition which creates a problem in the presence of many
> small partitions (which is becoming a common usecase in facebook).
> If the plan is the same, a mapper should be able to span files across multiple partitions
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HIVE-1900) a mapper should be able to span
multiple partitions
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-1900:
-----------------------------
Attachment: hive.1900.2.patch
> a mapper should be able to span multiple partitions
> ---------------------------------------------------
>
> Key: HIVE-1900
> URL: https://issues.apache.org/jira/browse/HIVE-1900
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Krishna Kumar
> Attachments: hive.1900.1.patch, hive.1900.2.patch
>
>
> Currently, a mapper only spans a single partition which creates a problem in the presence of many
> small partitions (which is becoming a common usecase in facebook).
> If the plan is the same, a mapper should be able to span files across multiple partitions
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1900) a mapper should be able to span
multiple partitions
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978918#action_12978918 ]
Ning Zhang commented on HIVE-1900:
----------------------------------
I remember I had encountered the problem before. Enabling a mapper to read from multiple partitions is trivial but there are some pitfalls to watch:
1) partitioning columns are not present in the data file itself. The partitioning column value is appended during the RecordReader (or something like that). It assumes that all records come from the same partition. The assumption will be broken here. An example query you can try is
select ds, count(1) from srcpart where ds is not null group by ds;
2) The merge job should be treated specially to not allow combined input from multiple partitions.
3) Auto-gathering stats from the FileSinkOperator need to be address for the case so that stats are maintained for multiple partitions.
> a mapper should be able to span multiple partitions
> ---------------------------------------------------
>
> Key: HIVE-1900
> URL: https://issues.apache.org/jira/browse/HIVE-1900
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: He Yongqiang
>
> Currently, a mapper only spans a single partition which creates a problem in the presence of many
> small partitions (which is becoming a common usecase in facebook).
> If the plan is the same, a mapper should be able to span files across multiple partitions
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1900) a mapper should be able to span
multiple partitions
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990673#comment-12990673 ]
Namit Jain commented on HIVE-1900:
----------------------------------
https://reviews.apache.org/r/393/
> a mapper should be able to span multiple partitions
> ---------------------------------------------------
>
> Key: HIVE-1900
> URL: https://issues.apache.org/jira/browse/HIVE-1900
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.1900.1.patch
>
>
> Currently, a mapper only spans a single partition which creates a problem in the presence of many
> small partitions (which is becoming a common usecase in facebook).
> If the plan is the same, a mapper should be able to span files across multiple partitions
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1900) a mapper should be able to span
multiple partitions
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978997#action_12978997 ]
Ning Zhang commented on HIVE-1900:
----------------------------------
Namit, do you mean bucketized sort-merge join? In that case don't you need to use a specialized InputFormat and RecordReader? If we allow mappers get inputs from multiple partitions, we need to ensure HiveInputFormat and CombineHiveInputFormat and the RecordReaders be partition aware.
2) is important because we don't want to merge different partitions in one file. Otherwise you need a dynamic partition insert for the merge which may generate multiple small files for a partition again.
3) If TableScanOperator can take multiple partitions, the stats has to be gathered according to the input partition column values. Currently the partition column value is checked for the 1st row and assumes all the rows have the same partitioning column value. If we allow multiple partitions in a mapper, we have to check the partition column values for each row.
> a mapper should be able to span multiple partitions
> ---------------------------------------------------
>
> Key: HIVE-1900
> URL: https://issues.apache.org/jira/browse/HIVE-1900
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: He Yongqiang
>
> Currently, a mapper only spans a single partition which creates a problem in the presence of many
> small partitions (which is becoming a common usecase in facebook).
> If the plan is the same, a mapper should be able to span files across multiple partitions
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-1900) a mapper should be able to span
multiple partitions
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain reassigned HIVE-1900:
--------------------------------
Assignee: Namit Jain (was: He Yongqiang)
> a mapper should be able to span multiple partitions
> ---------------------------------------------------
>
> Key: HIVE-1900
> URL: https://issues.apache.org/jira/browse/HIVE-1900
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
>
> Currently, a mapper only spans a single partition which creates a problem in the presence of many
> small partitions (which is becoming a common usecase in facebook).
> If the plan is the same, a mapper should be able to span files across multiple partitions
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1900) a mapper should be able to span
multiple partitions
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990796#comment-12990796 ]
He Yongqiang commented on HIVE-1900:
------------------------------------
running tests.
> a mapper should be able to span multiple partitions
> ---------------------------------------------------
>
> Key: HIVE-1900
> URL: https://issues.apache.org/jira/browse/HIVE-1900
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Krishna Kumar
> Attachments: hive.1900.1.patch, hive.1900.2.patch
>
>
> Currently, a mapper only spans a single partition which creates a problem in the presence of many
> small partitions (which is becoming a common usecase in facebook).
> If the plan is the same, a mapper should be able to span files across multiple partitions
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira