You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Siying Dong (JIRA)" <ji...@apache.org> on 2010/10/26 04:53:21 UTC
[jira] Created: (HIVE-1751) Optimize
ColumnarStructObjectInspector.getStructFieldData()
Optimize ColumnarStructObjectInspector.getStructFieldData()
-----------------------------------------------------------
Key: HIVE-1751
URL: https://issues.apache.org/jira/browse/HIVE-1751
Project: Hive
Issue Type: Improvement
Reporter: Siying Dong
Assignee: Siying Dong
ColumnarStructObjectInspector.getStructFieldData() is a heavy used function and is expensive.
By optimizing this function, including ColumnarStruct.uncheckedGetField() called by it, most queries can benefit from it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1751) Optimize
ColumnarStructObjectInspector.getStructFieldData()
Posted by "Siying Dong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siying Dong updated HIVE-1751:
------------------------------
Attachment: HIVE-1751.1.patch
ExprNodeColumnEvaluator.evaluate() is very heavily used function. For most queries, it is called multiple time per row. In the group-by query in the benchmark, it is even 10 times per row.
This function call sometimes takes 17%-20% CPU time. Usually ExprNodeColumnEvaluator.evaluate() itself takes 2%-3%, UnionStructObjectInspector.getStructFieldData() itself takes 2%-3%, ColumnarStruct.uncheckedGetField() itself takes 3%.
It's hard to come up with a general solution that reduce the costs in a structual way. I tried to did several small code rewriting and hope we can get slight improvements:
1. nullSequence is not passed in for every call but from constructor
2. Restructure ColumnarStruct a little bit.
3. In ExprNodeColumnEvaluator, makes the single level special case, which in most of the time is the common case when referring a column.
When trying to optimize functions which already only take 3%, it's hard to verify the performance enhancement since experiments anyway have slight variation eveyr time.
For 1 and 2, I think they anyway make code better readable. I ran many times, and consistently see about 1% improvement too.
3 might make code less readable, but I see about 5% improvement from some simple group-by query.
> Optimize ColumnarStructObjectInspector.getStructFieldData()
> -----------------------------------------------------------
>
> Key: HIVE-1751
> URL: https://issues.apache.org/jira/browse/HIVE-1751
> Project: Hive
> Issue Type: Improvement
> Reporter: Siying Dong
> Assignee: Siying Dong
> Attachments: HIVE-1751.1.patch
>
>
> ColumnarStructObjectInspector.getStructFieldData() is a heavy used function and is expensive.
> By optimizing this function, including ColumnarStruct.uncheckedGetField() called by it, most queries can benefit from it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1751) Optimize
ColumnarStructObjectInspector.getStructFieldData()
Posted by "Siying Dong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siying Dong updated HIVE-1751:
------------------------------
Status: Patch Available (was: Open)
> Optimize ColumnarStructObjectInspector.getStructFieldData()
> -----------------------------------------------------------
>
> Key: HIVE-1751
> URL: https://issues.apache.org/jira/browse/HIVE-1751
> Project: Hive
> Issue Type: Improvement
> Reporter: Siying Dong
> Assignee: Siying Dong
> Attachments: HIVE-1751.1.patch
>
>
> ColumnarStructObjectInspector.getStructFieldData() is a heavy used function and is expensive.
> By optimizing this function, including ColumnarStruct.uncheckedGetField() called by it, most queries can benefit from it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1751) Optimize
ColumnarStructObjectInspector.getStructFieldData()
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924828#action_12924828 ]
Ashutosh Chauhan commented on HIVE-1751:
----------------------------------------
Can you provide some info how you are planning to optimize it?
> Optimize ColumnarStructObjectInspector.getStructFieldData()
> -----------------------------------------------------------
>
> Key: HIVE-1751
> URL: https://issues.apache.org/jira/browse/HIVE-1751
> Project: Hive
> Issue Type: Improvement
> Reporter: Siying Dong
> Assignee: Siying Dong
>
> ColumnarStructObjectInspector.getStructFieldData() is a heavy used function and is expensive.
> By optimizing this function, including ColumnarStruct.uncheckedGetField() called by it, most queries can benefit from it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1751) Optimize
ColumnarStructObjectInspector.getStructFieldData()
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-1751:
-----------------------------
Resolution: Fixed
Fix Version/s: 0.7.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed. Thanks Siying
> Optimize ColumnarStructObjectInspector.getStructFieldData()
> -----------------------------------------------------------
>
> Key: HIVE-1751
> URL: https://issues.apache.org/jira/browse/HIVE-1751
> Project: Hive
> Issue Type: Improvement
> Reporter: Siying Dong
> Assignee: Siying Dong
> Fix For: 0.7.0
>
> Attachments: HIVE-1751.1.patch
>
>
> ColumnarStructObjectInspector.getStructFieldData() is a heavy used function and is expensive.
> By optimizing this function, including ColumnarStruct.uncheckedGetField() called by it, most queries can benefit from it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1751) Optimize
ColumnarStructObjectInspector.getStructFieldData()
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928133#action_12928133 ]
Namit Jain commented on HIVE-1751:
----------------------------------
+1
running tests.
> Optimize ColumnarStructObjectInspector.getStructFieldData()
> -----------------------------------------------------------
>
> Key: HIVE-1751
> URL: https://issues.apache.org/jira/browse/HIVE-1751
> Project: Hive
> Issue Type: Improvement
> Reporter: Siying Dong
> Assignee: Siying Dong
> Attachments: HIVE-1751.1.patch
>
>
> ColumnarStructObjectInspector.getStructFieldData() is a heavy used function and is expensive.
> By optimizing this function, including ColumnarStruct.uncheckedGetField() called by it, most queries can benefit from it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.