You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Namit Jain (JIRA)" <ji...@apache.org> on 2009/06/16 00:56:08 UTC
[jira] Created: (HIVE-562) join does not work well if there is a
very large skew in keys
join does not work well if there is a very large skew in keys
-------------------------------------------------------------
Key: HIVE-562
URL: https://issues.apache.org/jira/browse/HIVE-562
Project: Hadoop Hive
Issue Type: Improvement
Components: Query Processor
Affects Versions: 0.4.0
Reporter: Namit Jain
Only the last table is streamed in case of regular joins.
So, for any other table, or for any small table (in case of map-joins), if the number of values for a given key are very large, it does not scale.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-562) join does not work well if there is a
very large skew in keys
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768449#action_12768449 ]
Ning Zhang commented on HIVE-562:
---------------------------------
Can this be done by storing input (storage in JoinOperator.java and CommonJoinOperator.java) from one group to an external-storage-backed hash table (eg., HTree in JDBM as map-side join) rather than main memory hash table?
> join does not work well if there is a very large skew in keys
> -------------------------------------------------------------
>
> Key: HIVE-562
> URL: https://issues.apache.org/jira/browse/HIVE-562
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Affects Versions: 0.4.0
> Reporter: Namit Jain
>
> Only the last table is streamed in case of regular joins.
> So, for any other table, or for any small table (in case of map-joins), if the number of values for a given key are very large, it does not scale.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-562) join does not work well if there is a
very large skew in keys
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784590#action_12784590 ]
Namit Jain commented on HIVE-562:
---------------------------------
Had a discussion with Zheng, Paul, Raghu and Yongqiang on this. This is roughly the plan:
1. https://issues.apache.org/jira/browse/HIVE-963
2. https://issues.apache.org/jira/browse/HIVE-964
3. https://issues.apache.org/jira/browse/HIVE-965
> join does not work well if there is a very large skew in keys
> -------------------------------------------------------------
>
> Key: HIVE-562
> URL: https://issues.apache.org/jira/browse/HIVE-562
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Affects Versions: 0.4.0
> Reporter: Namit Jain
>
> Only the last table is streamed in case of regular joins.
> So, for any other table, or for any small table (in case of map-joins), if the number of values for a given key are very large, it does not scale.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-562) join does not work well if there is a
very large skew in keys
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719842#action_12719842 ]
Namit Jain commented on HIVE-562:
---------------------------------
It can be handled in a couple of ways -
1. create another level of indirection, and then swap those units independently. For eg, instead of storing 'N' values with a key 'k', store that 'k' maps to 'k1', 'k2'.....'kn'
and then read all keys when needed.
2. the query is rewritten to a bunch of independent queries - at some layer. So instead of joining L and S:
a. S is broken into S1..Sn
b. Join L with S1..Sn
c. Merge the above results.
This can be done by the compiler or at the query specification time.
> join does not work well if there is a very large skew in keys
> -------------------------------------------------------------
>
> Key: HIVE-562
> URL: https://issues.apache.org/jira/browse/HIVE-562
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Affects Versions: 0.4.0
> Reporter: Namit Jain
>
> Only the last table is streamed in case of regular joins.
> So, for any other table, or for any small table (in case of map-joins), if the number of values for a given key are very large, it does not scale.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-562) join does not work well if there is a
very large skew in keys
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768720#action_12768720 ]
Namit Jain commented on HIVE-562:
---------------------------------
That will also work - but we dont know the number of rows before hand for a given key.
We can have some hueristics, based on which, a persistent storage can be used
> join does not work well if there is a very large skew in keys
> -------------------------------------------------------------
>
> Key: HIVE-562
> URL: https://issues.apache.org/jira/browse/HIVE-562
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Affects Versions: 0.4.0
> Reporter: Namit Jain
>
> Only the last table is streamed in case of regular joins.
> So, for any other table, or for any small table (in case of map-joins), if the number of values for a given key are very large, it does not scale.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-562) join does not work well if there is a
very large skew in keys
Posted by "Jeff Hammerbacher (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768758#action_12768758 ]
Jeff Hammerbacher commented on HIVE-562:
----------------------------------------
It may be worth looking into how Pig solved this problem; see http://wiki.apache.org/pig/PigSkewedJoinSpec for their design.
> join does not work well if there is a very large skew in keys
> -------------------------------------------------------------
>
> Key: HIVE-562
> URL: https://issues.apache.org/jira/browse/HIVE-562
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Affects Versions: 0.4.0
> Reporter: Namit Jain
>
> Only the last table is streamed in case of regular joins.
> So, for any other table, or for any small table (in case of map-joins), if the number of values for a given key are very large, it does not scale.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.