You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Zheng Shao (JIRA)" <ji...@apache.org> on 2009/07/13 06:56:14 UTC
[jira] Created: (HIVE-627) Optimizer should only access RowSchema
(and not RowResolver)
Optimizer should only access RowSchema (and not RowResolver)
------------------------------------------------------------
Key: HIVE-627
URL: https://issues.apache.org/jira/browse/HIVE-627
Project: Hadoop Hive
Issue Type: Improvement
Reporter: Zheng Shao
The column pruner is accessing RowResolver a lot of times, for things like reverseLookup, and get(alias, column).
These are not necessary - we should not need to translate an internal name to (alias, column) and then translate back. We should be able to use internal name from one operator to the other, using RowSchema.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-627) Optimizer should only access RowSchema
(and not RowResolver)
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730561#action_12730561 ]
Zheng Shao commented on HIVE-627:
---------------------------------
Will we be able to clean up the whole OperatorCompile before optimization?
Also I feel that 1 and 2 can be done together but 3 can be separate by itself. We can do 1 and 2 in a separate issue.
The changes in 3 includes:
3.1. clean up opParseContext before we enter into optimization step;
3.2. make ColumnPruner use RowSchema instead of RowResolver.
> Optimizer should only access RowSchema (and not RowResolver)
> ------------------------------------------------------------
>
> Key: HIVE-627
> URL: https://issues.apache.org/jira/browse/HIVE-627
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Zheng Shao
>
> The column pruner is accessing RowResolver a lot of times, for things like reverseLookup, and get(alias, column).
> These are not necessary - we should not need to translate an internal name to (alias, column) and then translate back. We should be able to use internal name from one operator to the other, using RowSchema.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-627) Optimizer should only access RowSchema
(and not RowResolver)
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730495#action_12730495 ]
Zheng Shao commented on HIVE-627:
---------------------------------
Talked with Namit offline. Namit can add more details, but the synopsis is that we should be able to do it, and the best way is to wipe out the RowResolvers before we enter into the optimization phase.
> Optimizer should only access RowSchema (and not RowResolver)
> ------------------------------------------------------------
>
> Key: HIVE-627
> URL: https://issues.apache.org/jira/browse/HIVE-627
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Zheng Shao
>
> The column pruner is accessing RowResolver a lot of times, for things like reverseLookup, and get(alias, column).
> These are not necessary - we should not need to translate an internal name to (alias, column) and then translate back. We should be able to use internal name from one operator to the other, using RowSchema.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-627) Optimizer should only access RowSchema
(and not RowResolver)
Posted by "He Yongqiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730758#action_12730758 ]
He Yongqiang commented on HIVE-627:
-----------------------------------
So this issue intends to remove all the usages of RowResolver after compile finished? And let the optimizers do not use row resovler any more?
RowResolver is not only used in cp, but also used in ppd. Do we need to remove them all?
> Optimizer should only access RowSchema (and not RowResolver)
> ------------------------------------------------------------
>
> Key: HIVE-627
> URL: https://issues.apache.org/jira/browse/HIVE-627
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Zheng Shao
>
> The column pruner is accessing RowResolver a lot of times, for things like reverseLookup, and get(alias, column).
> These are not necessary - we should not need to translate an internal name to (alias, column) and then translate back. We should be able to use internal name from one operator to the other, using RowSchema.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-627) Optimizer should only access RowSchema
(and not RowResolver)
Posted by "Prasad Chakka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730949#action_12730949 ]
Prasad Chakka commented on HIVE-627:
------------------------------------
ppd uses rowresolve to figure out which alias that a particular column belongs to. without this it is tough to figure out whether an expression is a candidate for ppd or not. so this information (column -> alias) should be preserved in any data structure that replaces rowresolver. i believe rowschema doesn't have this information right now.
> Optimizer should only access RowSchema (and not RowResolver)
> ------------------------------------------------------------
>
> Key: HIVE-627
> URL: https://issues.apache.org/jira/browse/HIVE-627
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Zheng Shao
>
> The column pruner is accessing RowResolver a lot of times, for things like reverseLookup, and get(alias, column).
> These are not necessary - we should not need to translate an internal name to (alias, column) and then translate back. We should be able to use internal name from one operator to the other, using RowSchema.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-627) Optimizer should only access RowSchema
(and not RowResolver)
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730557#action_12730557 ]
Namit Jain commented on HIVE-627:
---------------------------------
We can do the following cleanups:
1. Include a new field: OperatorCompile, which is a transient field, in the Operator
2. This can include the RowResolver - we can get rid of the whole hashmap - opParseContext. If other operators need more compile time information, they can extend OperatorCompile.
3. All resolutions have already happened till the time optimizer is called. So, we don't need to do any name resolution at that time. However, in order to ensure that, we should just cleanup
the row resolver from OperatorCompile before calling the optimizer.
> Optimizer should only access RowSchema (and not RowResolver)
> ------------------------------------------------------------
>
> Key: HIVE-627
> URL: https://issues.apache.org/jira/browse/HIVE-627
> Project: Hadoop Hive
> Issue Type: Improvement
> Reporter: Zheng Shao
>
> The column pruner is accessing RowResolver a lot of times, for things like reverseLookup, and get(alias, column).
> These are not necessary - we should not need to translate an internal name to (alias, column) and then translate back. We should be able to use internal name from one operator to the other, using RowSchema.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.