You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Namit Jain (JIRA)" <ji...@apache.org> on 2010/12/06 08:02:11 UTC
[jira] Created: (HIVE-1830) mappers in group followed by joins may
die OOM
mappers in group followed by joins may die OOM
----------------------------------------------
Key: HIVE-1830
URL: https://issues.apache.org/jira/browse/HIVE-1830
Project: Hive
Issue Type: Bug
Reporter: Namit Jain
Assignee: Liyin Tang
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1830) mappers in group followed by joins may
die OOM
Posted by "Liyin Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Liyin Tang updated HIVE-1830:
-----------------------------
Attachment: hive-1830-5.patch
1) Remove the debug statements
2) Add the memory threshold to group by desc.
> mappers in group followed by joins may die OOM
> ----------------------------------------------
>
> Key: HIVE-1830
> URL: https://issues.apache.org/jira/browse/HIVE-1830
> Project: Hive
> Issue Type: Bug
> Reporter: Namit Jain
> Assignee: Liyin Tang
> Attachments: hive-1830-1.patch, hive-1830-2.patch, hive-1830-3.patch, hive-1830-4.patch, hive-1830-5.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1830) mappers in group followed by joins may
die OOM
Posted by "Liyin Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Liyin Tang updated HIVE-1830:
-----------------------------
Attachment: hive-1830-4.patch
1) Add more descriptions in the config file
2) Set the memory usage of hashtable sink op and group by op into their desc.
The memory usage is deterministic after compiling stage.
> mappers in group followed by joins may die OOM
> ----------------------------------------------
>
> Key: HIVE-1830
> URL: https://issues.apache.org/jira/browse/HIVE-1830
> Project: Hive
> Issue Type: Bug
> Reporter: Namit Jain
> Assignee: Liyin Tang
> Attachments: hive-1830-1.patch, hive-1830-2.patch, hive-1830-3.patch, hive-1830-4.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-1830) mappers in group followed by joins may
die OOM
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain resolved HIVE-1830.
------------------------------
Resolution: Fixed
Fix Version/s: 0.7.0
Hadoop Flags: [Reviewed]
Committed. Thanks Liyin
> mappers in group followed by joins may die OOM
> ----------------------------------------------
>
> Key: HIVE-1830
> URL: https://issues.apache.org/jira/browse/HIVE-1830
> Project: Hive
> Issue Type: Bug
> Reporter: Namit Jain
> Assignee: Liyin Tang
> Fix For: 0.7.0
>
> Attachments: hive-1830-1.patch, hive-1830-2.patch, hive-1830-3.patch, hive-1830-4.patch, hive-1830-5.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1830) mappers in group followed by joins may
die OOM
Posted by "Liyin Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Liyin Tang updated HIVE-1830:
-----------------------------
Attachment: hive-1830-3.patch
Carefully measure the memory usage of map side group by.
Flush frequently, if the left memory is less than a threshold.
> mappers in group followed by joins may die OOM
> ----------------------------------------------
>
> Key: HIVE-1830
> URL: https://issues.apache.org/jira/browse/HIVE-1830
> Project: Hive
> Issue Type: Bug
> Reporter: Namit Jain
> Assignee: Liyin Tang
> Attachments: hive-1830-1.patch, hive-1830-2.patch, hive-1830-3.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1830) mappers in group followed by joins
may die OOM
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12967105#action_12967105 ]
Namit Jain commented on HIVE-1830:
----------------------------------
After HIVE-1642, joins are automatically converted into map-joins at physical optimization time.
However, this may lead to problems.
For eg: consider the query:
select T1.val, count(1) from T1 join T2 on T1.key=T2.key group by T1.val
This will have 2 map-reduce jobs, one for the join and the other for group by.
Before HIVE-1642, the partial group for aggregation will be performed in the reducer where the join is performed.
However, after HIVE-1642, the same will be performed in the mapper. The local task will confirm that there is just
enough memory to hold the map-join data. Hoever, it does not take into account the memory needed for partial group
by.
So, in case there is group by followed by join, it is a good idea to reduce the memory given to the local task to validate
if there is enough memory to fit small table - it can be controlled by a new configuration paramter, but it can be some
default: say 70% of total memory (instead of 90%).
Also, the group by may still run out of memory, so it might be a good idea to check in group by for free memory and
periodically flush memory
> mappers in group followed by joins may die OOM
> ----------------------------------------------
>
> Key: HIVE-1830
> URL: https://issues.apache.org/jira/browse/HIVE-1830
> Project: Hive
> Issue Type: Bug
> Reporter: Namit Jain
> Assignee: Liyin Tang
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1830) mappers in group followed by joins
may die OOM
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12969070#action_12969070 ]
Namit Jain commented on HIVE-1830:
----------------------------------
Add the new parameters description in hive-default.xml.
Move the memory threshold in the descriptors (for eg. all the needed confs should be copied in groupbydesc.
and accessed from there, instead of being accessed from HiveConf at runtime) - you already did it from
HashTableSinkOperator
> mappers in group followed by joins may die OOM
> ----------------------------------------------
>
> Key: HIVE-1830
> URL: https://issues.apache.org/jira/browse/HIVE-1830
> Project: Hive
> Issue Type: Bug
> Reporter: Namit Jain
> Assignee: Liyin Tang
> Attachments: hive-1830-1.patch, hive-1830-2.patch, hive-1830-3.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1830) mappers in group followed by joins may
die OOM
Posted by "Liyin Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Liyin Tang updated HIVE-1830:
-----------------------------
Attachment: hive-1830-2.patch
Add a new test: auto_join26.q
> mappers in group followed by joins may die OOM
> ----------------------------------------------
>
> Key: HIVE-1830
> URL: https://issues.apache.org/jira/browse/HIVE-1830
> Project: Hive
> Issue Type: Bug
> Reporter: Namit Jain
> Assignee: Liyin Tang
> Attachments: hive-1830-1.patch, hive-1830-2.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1830) mappers in group followed by joins
may die OOM
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12969549#action_12969549 ]
Namit Jain commented on HIVE-1830:
----------------------------------
+1
> mappers in group followed by joins may die OOM
> ----------------------------------------------
>
> Key: HIVE-1830
> URL: https://issues.apache.org/jira/browse/HIVE-1830
> Project: Hive
> Issue Type: Bug
> Reporter: Namit Jain
> Assignee: Liyin Tang
> Attachments: hive-1830-1.patch, hive-1830-2.patch, hive-1830-3.patch, hive-1830-4.patch, hive-1830-5.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1830) mappers in group followed by joins
may die OOM
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12969203#action_12969203 ]
Namit Jain commented on HIVE-1830:
----------------------------------
if (groupByOp.getConf() == null) {
91 System.out.println("Group by desc is null");
92 return null;
93 }
This should never happen
GroupByOperator:
memoryThreshold = HiveConf.getFloatVar(hconf, HiveConf.ConfVars.HIVEMAPAGGRM⬅
EMORYTHRESHOLD);
This should also be in groupByDesc
> mappers in group followed by joins may die OOM
> ----------------------------------------------
>
> Key: HIVE-1830
> URL: https://issues.apache.org/jira/browse/HIVE-1830
> Project: Hive
> Issue Type: Bug
> Reporter: Namit Jain
> Assignee: Liyin Tang
> Attachments: hive-1830-1.patch, hive-1830-2.patch, hive-1830-3.patch, hive-1830-4.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1830) mappers in group followed by joins
may die OOM
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12969061#action_12969061 ]
Namit Jain commented on HIVE-1830:
----------------------------------
I will take a look
> mappers in group followed by joins may die OOM
> ----------------------------------------------
>
> Key: HIVE-1830
> URL: https://issues.apache.org/jira/browse/HIVE-1830
> Project: Hive
> Issue Type: Bug
> Reporter: Namit Jain
> Assignee: Liyin Tang
> Attachments: hive-1830-1.patch, hive-1830-2.patch, hive-1830-3.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1830) mappers in group followed by joins may
die OOM
Posted by "Liyin Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Liyin Tang updated HIVE-1830:
-----------------------------
Attachment: hive-1830-1.patch
> mappers in group followed by joins may die OOM
> ----------------------------------------------
>
> Key: HIVE-1830
> URL: https://issues.apache.org/jira/browse/HIVE-1830
> Project: Hive
> Issue Type: Bug
> Reporter: Namit Jain
> Assignee: Liyin Tang
> Attachments: hive-1830-1.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.