You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Namit Jain (JIRA)" <ji...@apache.org> on 2010/10/28 22:15:20 UTC
[jira] Created: (HIVE-1758) optimize group by hash map memory
optimize group by hash map memory
---------------------------------
Key: HIVE-1758
URL: https://issues.apache.org/jira/browse/HIVE-1758
Project: Hive
Issue Type: Improvement
Components: Query Processor
Reporter: Namit Jain
Assignee: Siying Dong
Group By map side's hash map consumes a lot of memory, thereby decreasing its effectiveness.
We can use some of the optimizations from map-join to reduce the memory footprint:
class KeyWrapper {
int hashcode;
ArrayList<Object> keys;
// decide whether this is already in hashmap (keys in hashmap are deepcopied
// version, and we need to use 'currentKeyObjectInspector').
boolean copy = false;
1. Changes keys to Array
2. Optimize the scenario when keys is of a small size (1,2) etc
Let us start profiling it and take it from there
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1758) optimize group by hash map memory
Posted by "Siying Dong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siying Dong updated HIVE-1758:
------------------------------
Attachment: HIVE-1758.1.patch
1. use array instead of ArrayList<Object>
2. abstract KeyWrapper a little bit so that we can write specific codes for different kind of keys
3. use ListKeyWrapper and TestKeyWrapper
The motivation of the task is to reduce memory footprint. There is minor impact to CPU time.
> optimize group by hash map memory
> ---------------------------------
>
> Key: HIVE-1758
> URL: https://issues.apache.org/jira/browse/HIVE-1758
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Siying Dong
> Attachments: HIVE-1758.1.patch
>
>
> Group By map side's hash map consumes a lot of memory, thereby decreasing its effectiveness.
> We can use some of the optimizations from map-join to reduce the memory footprint:
> class KeyWrapper {
> int hashcode;
> ArrayList<Object> keys;
> // decide whether this is already in hashmap (keys in hashmap are deepcopied
> // version, and we need to use 'currentKeyObjectInspector').
> boolean copy = false;
> 1. Changes keys to Array
> 2. Optimize the scenario when keys is of a small size (1,2) etc
> Let us start profiling it and take it from there
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1758) optimize group by hash map memory
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-1758:
-----------------------------
Resolution: Fixed
Fix Version/s: 0.7.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed. Thanks Siying
> optimize group by hash map memory
> ---------------------------------
>
> Key: HIVE-1758
> URL: https://issues.apache.org/jira/browse/HIVE-1758
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Siying Dong
> Fix For: 0.7.0
>
> Attachments: HIVE-1758.1.patch
>
>
> Group By map side's hash map consumes a lot of memory, thereby decreasing its effectiveness.
> We can use some of the optimizations from map-join to reduce the memory footprint:
> class KeyWrapper {
> int hashcode;
> ArrayList<Object> keys;
> // decide whether this is already in hashmap (keys in hashmap are deepcopied
> // version, and we need to use 'currentKeyObjectInspector').
> boolean copy = false;
> 1. Changes keys to Array
> 2. Optimize the scenario when keys is of a small size (1,2) etc
> Let us start profiling it and take it from there
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1758) optimize group by hash map memory
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930918#action_12930918 ]
Namit Jain commented on HIVE-1758:
----------------------------------
+1
Looks good and also merges well after the other optimization HIVE-1743
It would be good to have a keywrapper for the case of int, 2 strings also but that can be done in a followup.
> optimize group by hash map memory
> ---------------------------------
>
> Key: HIVE-1758
> URL: https://issues.apache.org/jira/browse/HIVE-1758
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Siying Dong
> Attachments: HIVE-1758.1.patch
>
>
> Group By map side's hash map consumes a lot of memory, thereby decreasing its effectiveness.
> We can use some of the optimizations from map-join to reduce the memory footprint:
> class KeyWrapper {
> int hashcode;
> ArrayList<Object> keys;
> // decide whether this is already in hashmap (keys in hashmap are deepcopied
> // version, and we need to use 'currentKeyObjectInspector').
> boolean copy = false;
> 1. Changes keys to Array
> 2. Optimize the scenario when keys is of a small size (1,2) etc
> Let us start profiling it and take it from there
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1758) optimize group by hash map memory
Posted by "Siying Dong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siying Dong updated HIVE-1758:
------------------------------
Status: Patch Available (was: Open)
> optimize group by hash map memory
> ---------------------------------
>
> Key: HIVE-1758
> URL: https://issues.apache.org/jira/browse/HIVE-1758
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Siying Dong
> Attachments: HIVE-1758.1.patch
>
>
> Group By map side's hash map consumes a lot of memory, thereby decreasing its effectiveness.
> We can use some of the optimizations from map-join to reduce the memory footprint:
> class KeyWrapper {
> int hashcode;
> ArrayList<Object> keys;
> // decide whether this is already in hashmap (keys in hashmap are deepcopied
> // version, and we need to use 'currentKeyObjectInspector').
> boolean copy = false;
> 1. Changes keys to Array
> 2. Optimize the scenario when keys is of a small size (1,2) etc
> Let us start profiling it and take it from there
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.