You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Namit Jain (JIRA)" <ji...@apache.org> on 2009/12/02 03:03:20 UTC
[jira] Created: (HIVE-963) no out of memory errors for skewed join
no out of memory errors for skewed join
---------------------------------------
Key: HIVE-963
URL: https://issues.apache.org/jira/browse/HIVE-963
Project: Hadoop Hive
Issue Type: Improvement
Components: Query Processor
Reporter: Namit Jain
Assignee: He Yongqiang
Currently, in case of skew, hive runs out of memory.
A simpler fix would be to use JDBM to store data and use that.
It can be configurable and JDBM should only be triggered if the number of values for a given key exceed a given number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-963) no out of memory errors for skewed
join
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791228#action_12791228 ]
Namit Jain commented on HIVE-963:
---------------------------------
I have take a look and get back to you
> no out of memory errors for skewed join
> ---------------------------------------
>
> Key: HIVE-963
> URL: https://issues.apache.org/jira/browse/HIVE-963
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Ning Zhang
> Attachments: HIVE-963.patch
>
>
> Currently, in case of skew, hive runs out of memory.
> A simpler fix would be to use JDBM to store data and use that.
> It can be configurable and JDBM should only be triggered if the number of values for a given key exceed a given number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-963) no out of memory errors for skewed
join
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12786100#action_12786100 ]
Ning Zhang commented on HIVE-963:
---------------------------------
Found some issues in the case of CommonJoin. The # of keys here is small but the value is too large. So this implementation won't help much. It will help for the MapJoin case. Please ignore this patch for now.
> no out of memory errors for skewed join
> ---------------------------------------
>
> Key: HIVE-963
> URL: https://issues.apache.org/jira/browse/HIVE-963
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Ning Zhang
> Attachments: HIVE-963.patch
>
>
> Currently, in case of skew, hive runs out of memory.
> A simpler fix would be to use JDBM to store data and use that.
> It can be configurable and JDBM should only be triggered if the number of values for a given key exceed a given number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-963) no out of memory errors for skewed
join
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791238#action_12791238 ]
Ning Zhang commented on HIVE-963:
---------------------------------
good point. Actually I had that code in startGroup() before but it introduced a bug when one of ArrayListWrapper is a dummyObj, in which case we don't want to clear it. I agree that if it is not dummyObj, alw should be cleared (or should be reused).
> no out of memory errors for skewed join
> ---------------------------------------
>
> Key: HIVE-963
> URL: https://issues.apache.org/jira/browse/HIVE-963
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Ning Zhang
> Attachments: HIVE-963.patch
>
>
> Currently, in case of skew, hive runs out of memory.
> A simpler fix would be to use JDBM to store data and use that.
> It can be configurable and JDBM should only be triggered if the number of values for a given key exceed a given number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-963) no out of memory errors for skewed join
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang updated HIVE-963:
----------------------------
Attachment: HIVE-963.patch
HIVE-963.patch is attached. This patch
1) introduced a new file HashMapWrapper.java
2) changed JoinOperator, CommonJoinOperator, and MapJoinOperator to use HashMapWrapper.
3) modified JDBM files to add support to disable cache and accepting File as the input parameter to record manager.
> no out of memory errors for skewed join
> ---------------------------------------
>
> Key: HIVE-963
> URL: https://issues.apache.org/jira/browse/HIVE-963
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Ning Zhang
> Attachments: HIVE-963.patch
>
>
> Currently, in case of skew, hive runs out of memory.
> A simpler fix would be to use JDBM to store data and use that.
> It can be configurable and JDBM should only be triggered if the number of values for a given key exceed a given number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-963) no out of memory errors for skewed
join
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791231#action_12791231 ]
Namit Jain commented on HIVE-963:
---------------------------------
looks good - 1 minor comment.
Dont you need to clean up the temp files at the beginning of each group.
for ( ArrayListWrapper alw: storage.values() )
alw.clear(); // clean up the temp files
storage.clear();
Isn't this needed in startGroup() also ?
> no out of memory errors for skewed join
> ---------------------------------------
>
> Key: HIVE-963
> URL: https://issues.apache.org/jira/browse/HIVE-963
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Ning Zhang
> Attachments: HIVE-963.patch
>
>
> Currently, in case of skew, hive runs out of memory.
> A simpler fix would be to use JDBM to store data and use that.
> It can be configurable and JDBM should only be triggered if the number of values for a given key exceed a given number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-963) no out of memory errors for skewed
join
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12785684#action_12785684 ]
Namit Jain commented on HIVE-963:
---------------------------------
In JDBM, the cache can be disabled very easily
> no out of memory errors for skewed join
> ---------------------------------------
>
> Key: HIVE-963
> URL: https://issues.apache.org/jira/browse/HIVE-963
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Ning Zhang
>
> Currently, in case of skew, hive runs out of memory.
> A simpler fix would be to use JDBM to store data and use that.
> It can be configurable and JDBM should only be triggered if the number of values for a given key exceed a given number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-963) no out of memory errors for skewed
join
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12785687#action_12785687 ]
Raghotham Murthy commented on HIVE-963:
---------------------------------------
Actually, after thinking about this a little more, we dont need a hashmap or jdbm for skewed joins. We run out of memory in the reducer when data is skewed. And in the reducer, data is already sorted.
So, we just need to read the first table's data into an array and when we run out of memory, just dump the remaining data into a tmp file. Then, while streaming through the second table, we first join with the data in memory and then swap the contents of the tmp file for the rest of the join. We could optimize by reading in several tuples from the second table into memory instead of processing one tuple at a time.
does that make sense?
> no out of memory errors for skewed join
> ---------------------------------------
>
> Key: HIVE-963
> URL: https://issues.apache.org/jira/browse/HIVE-963
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Ning Zhang
>
> Currently, in case of skew, hive runs out of memory.
> A simpler fix would be to use JDBM to store data and use that.
> It can be configurable and JDBM should only be triggered if the number of values for a given key exceed a given number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-963) no out of memory errors for skewed
join
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12785691#action_12785691 ]
Namit Jain commented on HIVE-963:
---------------------------------
Yes, that makes sense. I think 2 discussions are happening on this jira - in general it is a good idea to have a cache on top of JDBM, and not write every row to a file.
That should speed up the mapper considerably for map-joins.
However, for a join happening on the reducer, JDBM seems too heavy weight to use.
> no out of memory errors for skewed join
> ---------------------------------------
>
> Key: HIVE-963
> URL: https://issues.apache.org/jira/browse/HIVE-963
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Ning Zhang
>
> Currently, in case of skew, hive runs out of memory.
> A simpler fix would be to use JDBM to store data and use that.
> It can be configurable and JDBM should only be triggered if the number of values for a given key exceed a given number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-963) no out of memory errors for skewed join
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang reassigned HIVE-963:
-------------------------------
Assignee: Ning Zhang (was: He Yongqiang)
> no out of memory errors for skewed join
> ---------------------------------------
>
> Key: HIVE-963
> URL: https://issues.apache.org/jira/browse/HIVE-963
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Ning Zhang
>
> Currently, in case of skew, hive runs out of memory.
> A simpler fix would be to use JDBM to store data and use that.
> It can be configurable and JDBM should only be triggered if the number of values for a given key exceed a given number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-963) no out of memory errors for skewed join
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-963:
----------------------------
Resolution: Fixed
Fix Version/s: 0.5.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed. Thanks Ning
> no out of memory errors for skewed join
> ---------------------------------------
>
> Key: HIVE-963
> URL: https://issues.apache.org/jira/browse/HIVE-963
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Ning Zhang
> Fix For: 0.5.0
>
> Attachments: HIVE-963.patch, HIVE-963_2.patch, HIVE-963_3.patch
>
>
> Currently, in case of skew, hive runs out of memory.
> A simpler fix would be to use JDBM to store data and use that.
> It can be configurable and JDBM should only be triggered if the number of values for a given key exceed a given number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-963) no out of memory errors for skewed join
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang updated HIVE-963:
----------------------------
Attachment: HIVE-963.patch
Uploading patch HIVE-963.patch. This patch include:
1) Introducing a new class ArrayListWrapper that implements a simple persistent ArrayList interface (subset). It tries to solve two issues: OutOfMemoryException and data loading and iteration performance. It only supports add() and a fast iterator iterface through fist() and next(). So no Iterator class is created. Preliminary experiments shows a 10x faster in data load.
2) Changed MapJoinOperator, CommonJoinOperator to use ArrayListWrapper.
3) add a new unit test TestArrayListWrapper.
4) misc. simple factoring and code cleanup in HashMapWrapper.
> no out of memory errors for skewed join
> ---------------------------------------
>
> Key: HIVE-963
> URL: https://issues.apache.org/jira/browse/HIVE-963
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Ning Zhang
> Attachments: HIVE-963.patch
>
>
> Currently, in case of skew, hive runs out of memory.
> A simpler fix would be to use JDBM to store data and use that.
> It can be configurable and JDBM should only be triggered if the number of values for a given key exceed a given number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-963) no out of memory errors for skewed join
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang updated HIVE-963:
----------------------------
Status: Patch Available (was: Open)
> no out of memory errors for skewed join
> ---------------------------------------
>
> Key: HIVE-963
> URL: https://issues.apache.org/jira/browse/HIVE-963
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Ning Zhang
> Attachments: HIVE-963.patch
>
>
> Currently, in case of skew, hive runs out of memory.
> A simpler fix would be to use JDBM to store data and use that.
> It can be configurable and JDBM should only be triggered if the number of values for a given key exceed a given number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-963) no out of memory errors for skewed
join
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12785680#action_12785680 ]
Raghotham Murthy commented on HIVE-963:
---------------------------------------
wont we have to allocate some memory to jdbm as well? Will we have enough memory for both the hashmap and jdbm?
> no out of memory errors for skewed join
> ---------------------------------------
>
> Key: HIVE-963
> URL: https://issues.apache.org/jira/browse/HIVE-963
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Ning Zhang
>
> Currently, in case of skew, hive runs out of memory.
> A simpler fix would be to use JDBM to store data and use that.
> It can be configurable and JDBM should only be triggered if the number of values for a given key exceed a given number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-963) no out of memory errors for skewed
join
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784626#action_12784626 ]
Ning Zhang commented on HIVE-963:
---------------------------------
I think we can define a class implementing the Map interface as a wrapper of HashMap and JDBM HTree, so that map-side join can also use this class rather than using JDBM unconditionally. It is also easier if we decided to replace JDBM with a simpler and more efficient persistent hash implementation.
> no out of memory errors for skewed join
> ---------------------------------------
>
> Key: HIVE-963
> URL: https://issues.apache.org/jira/browse/HIVE-963
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: He Yongqiang
>
> Currently, in case of skew, hive runs out of memory.
> A simpler fix would be to use JDBM to store data and use that.
> It can be configurable and JDBM should only be triggered if the number of values for a given key exceed a given number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-963) no out of memory errors for skewed
join
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794985#action_12794985 ]
Namit Jain commented on HIVE-963:
---------------------------------
+1
looks good - will commit if the tests pass
> no out of memory errors for skewed join
> ---------------------------------------
>
> Key: HIVE-963
> URL: https://issues.apache.org/jira/browse/HIVE-963
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Ning Zhang
> Attachments: HIVE-963.patch, HIVE-963_2.patch, HIVE-963_3.patch
>
>
> Currently, in case of skew, hive runs out of memory.
> A simpler fix would be to use JDBM to store data and use that.
> It can be configurable and JDBM should only be triggered if the number of values for a given key exceed a given number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-963) no out of memory errors for skewed
join
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12785659#action_12785659 ]
Raghotham Murthy commented on HIVE-963:
---------------------------------------
its better (performance-wise) to extend jdbm instead of copying over records from the hashmap to jdbm if some threshold is exceeded.
> no out of memory errors for skewed join
> ---------------------------------------
>
> Key: HIVE-963
> URL: https://issues.apache.org/jira/browse/HIVE-963
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Ning Zhang
>
> Currently, in case of skew, hive runs out of memory.
> A simpler fix would be to use JDBM to store data and use that.
> It can be configurable and JDBM should only be triggered if the number of values for a given key exceed a given number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-963) no out of memory errors for skewed
join
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794940#action_12794940 ]
Namit Jain commented on HIVE-963:
---------------------------------
1. Can you remove Row.java
2. Can you also do some refactoring - move RowContainer/MapJoinObjectKey and MapJoinObjectValue in a separate directory in exec (joinUtilities or something like that?)
> no out of memory errors for skewed join
> ---------------------------------------
>
> Key: HIVE-963
> URL: https://issues.apache.org/jira/browse/HIVE-963
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Ning Zhang
> Attachments: HIVE-963.patch, HIVE-963_2.patch
>
>
> Currently, in case of skew, hive runs out of memory.
> A simpler fix would be to use JDBM to store data and use that.
> It can be configurable and JDBM should only be triggered if the number of values for a given key exceed a given number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-963) no out of memory errors for skewed
join
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12785689#action_12785689 ]
Ning Zhang commented on HIVE-963:
---------------------------------
Yes, in our application cache in JDBM can be disabled. The memory footprint for JDBM should be minimum, which will be needed anyway if we add cache layer to JDBM directly.
> no out of memory errors for skewed join
> ---------------------------------------
>
> Key: HIVE-963
> URL: https://issues.apache.org/jira/browse/HIVE-963
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Ning Zhang
>
> Currently, in case of skew, hive runs out of memory.
> A simpler fix would be to use JDBM to store data and use that.
> It can be configurable and JDBM should only be triggered if the number of values for a given key exceed a given number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-963) no out of memory errors for skewed
join
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784665#action_12784665 ]
Namit Jain commented on HIVE-963:
---------------------------------
We can also extend JDBM to spill only when needed.
+1 to the idea of having a separate class
> no out of memory errors for skewed join
> ---------------------------------------
>
> Key: HIVE-963
> URL: https://issues.apache.org/jira/browse/HIVE-963
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: He Yongqiang
>
> Currently, in case of skew, hive runs out of memory.
> A simpler fix would be to use JDBM to store data and use that.
> It can be configurable and JDBM should only be triggered if the number of values for a given key exceed a given number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-963) no out of memory errors for skewed join
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang updated HIVE-963:
----------------------------
Attachment: HIVE-963_2.patch
Uploading a new patch. Changes include:
1) change ArrayListWrapper to RowContainer.
2) In RowContainer, the input shoudl always be a list of object, which should be come with a SerDe and ObjectInspector to spill out to disk.
3) added in the joinDesc to indicate the tableDesc of the inputs to the join operator.
4) added a union test join40.q to test the case when the block size is 1 where spill will actually happen.
> no out of memory errors for skewed join
> ---------------------------------------
>
> Key: HIVE-963
> URL: https://issues.apache.org/jira/browse/HIVE-963
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Ning Zhang
> Attachments: HIVE-963.patch, HIVE-963_2.patch
>
>
> Currently, in case of skew, hive runs out of memory.
> A simpler fix would be to use JDBM to store data and use that.
> It can be configurable and JDBM should only be triggered if the number of values for a given key exceed a given number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-963) no out of memory errors for skewed join
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang updated HIVE-963:
----------------------------
Attachment: HIVE-963_3.patch
HIVE-963_3.patch contains the changes suggested by Namit previous comments.
> no out of memory errors for skewed join
> ---------------------------------------
>
> Key: HIVE-963
> URL: https://issues.apache.org/jira/browse/HIVE-963
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Ning Zhang
> Attachments: HIVE-963.patch, HIVE-963_2.patch, HIVE-963_3.patch
>
>
> Currently, in case of skew, hive runs out of memory.
> A simpler fix would be to use JDBM to store data and use that.
> It can be configurable and JDBM should only be triggered if the number of values for a given key exceed a given number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-963) no out of memory errors for skewed
join
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791434#action_12791434 ]
Namit Jain commented on HIVE-963:
---------------------------------
Also, can you add a test or convert an existing one with join.size = 1, so that the persistent code is executed
> no out of memory errors for skewed join
> ---------------------------------------
>
> Key: HIVE-963
> URL: https://issues.apache.org/jira/browse/HIVE-963
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Ning Zhang
> Attachments: HIVE-963.patch
>
>
> Currently, in case of skew, hive runs out of memory.
> A simpler fix would be to use JDBM to store data and use that.
> It can be configurable and JDBM should only be triggered if the number of values for a given key exceed a given number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-963) no out of memory errors for skewed
join
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12785443#action_12785443 ]
Ning Zhang commented on HIVE-963:
---------------------------------
Yongqiang, have you started this JIRA. If not, do you mind I taking over this JIRA? There is a task that was blocked by this JIRA.
> no out of memory errors for skewed join
> ---------------------------------------
>
> Key: HIVE-963
> URL: https://issues.apache.org/jira/browse/HIVE-963
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: He Yongqiang
>
> Currently, in case of skew, hive runs out of memory.
> A simpler fix would be to use JDBM to store data and use that.
> It can be configurable and JDBM should only be triggered if the number of values for a given key exceed a given number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-963) no out of memory errors for skewed
join
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12785447#action_12785447 ]
He Yongqiang commented on HIVE-963:
-----------------------------------
no, i haven't start working on it. Please feel free to take over.
> no out of memory errors for skewed join
> ---------------------------------------
>
> Key: HIVE-963
> URL: https://issues.apache.org/jira/browse/HIVE-963
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: He Yongqiang
>
> Currently, in case of skew, hive runs out of memory.
> A simpler fix would be to use JDBM to store data and use that.
> It can be configurable and JDBM should only be triggered if the number of values for a given key exceed a given number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-963) no out of memory errors for skewed
join
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791219#action_12791219 ]
Ning Zhang commented on HIVE-963:
---------------------------------
deleted the old HIVE-963.patch. The current HIVE-963.patch is the new one with all the changes mentioned above. It is ready for review.
> no out of memory errors for skewed join
> ---------------------------------------
>
> Key: HIVE-963
> URL: https://issues.apache.org/jira/browse/HIVE-963
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Ning Zhang
> Attachments: HIVE-963.patch
>
>
> Currently, in case of skew, hive runs out of memory.
> A simpler fix would be to use JDBM to store data and use that.
> It can be configurable and JDBM should only be triggered if the number of values for a given key exceed a given number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-963) no out of memory errors for skewed
join
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12785663#action_12785663 ]
Ning Zhang commented on HIVE-963:
---------------------------------
We don't need to copy hashmap to jdbm. We'll keep data in HashMap and only put new data in JDBM if overflow. That's easier than modifying JDBM to enable cache w/o writing to disk as well.
> no out of memory errors for skewed join
> ---------------------------------------
>
> Key: HIVE-963
> URL: https://issues.apache.org/jira/browse/HIVE-963
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Ning Zhang
>
> Currently, in case of skew, hive runs out of memory.
> A simpler fix would be to use JDBM to store data and use that.
> It can be configurable and JDBM should only be triggered if the number of values for a given key exceed a given number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-963) no out of memory errors for skewed join
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang updated HIVE-963:
----------------------------
Attachment: (was: HIVE-963.patch)
> no out of memory errors for skewed join
> ---------------------------------------
>
> Key: HIVE-963
> URL: https://issues.apache.org/jira/browse/HIVE-963
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Ning Zhang
> Attachments: HIVE-963.patch
>
>
> Currently, in case of skew, hive runs out of memory.
> A simpler fix would be to use JDBM to store data and use that.
> It can be configurable and JDBM should only be triggered if the number of values for a given key exceed a given number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-963) no out of memory errors for skewed
join
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12785692#action_12785692 ]
Ning Zhang commented on HIVE-963:
---------------------------------
@ragu, I think what you proposed is the similar to what Zheng proposed to implement a lightweight persistent hash table rather than using JDBM. I think that can be done in the second stage (the next JIRA?). We can replace the JDBM in the wrapper by the lightweight persistent hash table.
Basically we need the hash table in the case of joinning of multiple tables, where the key is the table tag. Of couse hash map can be avoided by using list of arrays, if we know the table tags are always small integers.
> no out of memory errors for skewed join
> ---------------------------------------
>
> Key: HIVE-963
> URL: https://issues.apache.org/jira/browse/HIVE-963
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Ning Zhang
>
> Currently, in case of skew, hive runs out of memory.
> A simpler fix would be to use JDBM to store data and use that.
> It can be configurable and JDBM should only be triggered if the number of values for a given key exceed a given number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.