You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Jonathan Chang (Created) (JIRA)" <ji...@apache.org> on 2011/11/17 22:45:57 UTC
[jira] [Created] (HIVE-2592) ConstantOI doesn't always propagate to
UDAF
ConstantOI doesn't always propagate to UDAF
-------------------------------------------
Key: HIVE-2592
URL: https://issues.apache.org/jira/browse/HIVE-2592
Project: Hive
Issue Type: Bug
Reporter: Jonathan Chang
Assignee: Jonathan Chang
Depending on the query plan, a UDAF may not actually get a constant OI. E.g.,
set hive.map.aggr=false;
SELECT percentile_approx(cast(substr(src.value,5) AS double), 0.5) FROM src;
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2592) ConstantOI doesn't always propagate
to UDAF
Posted by "Jonathan Chang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13153644#comment-13153644 ]
Jonathan Chang commented on HIVE-2592:
--------------------------------------
@jvs, I have a rather large diff that goes some of the way towards fixing the issue, but it will take a few more large diffs before this one is knocked down completely. Would you be amenable to reviewing/committing one of them?
> ConstantOI doesn't always propagate to UDAF
> -------------------------------------------
>
> Key: HIVE-2592
> URL: https://issues.apache.org/jira/browse/HIVE-2592
> Project: Hive
> Issue Type: Bug
> Reporter: Jonathan Chang
> Assignee: Jonathan Chang
>
> Depending on the query plan, a UDAF may not actually get a constant OI. E.g.,
> set hive.map.aggr=false;
> SELECT percentile_approx(cast(substr(src.value,5) AS double), 0.5) FROM src;
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2592) ConstantOI doesn't always propagate
to UDAF
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155269#comment-13155269 ]
Phabricator commented on HIVE-2592:
-----------------------------------
jonchang has commented on the revision "HIVE-2592 [jira] ConstantOI doesn't always propagate to UDAF".
I dunno if there are any valid uses of TypeInfo.
Anyhow, this diff gets part of the way towards fixing the other issue (constant OI into partial aggregates), but the serialization still needs to be fixed.
REVISION DETAIL
https://reviews.facebook.net/D513
> ConstantOI doesn't always propagate to UDAF
> -------------------------------------------
>
> Key: HIVE-2592
> URL: https://issues.apache.org/jira/browse/HIVE-2592
> Project: Hive
> Issue Type: Bug
> Reporter: Jonathan Chang
> Assignee: Jonathan Chang
> Attachments: HIVE-2592.D513.1.patch, HIVE-2592.D513.2.patch
>
>
> Depending on the query plan, a UDAF may not actually get a constant OI. E.g.,
> set hive.map.aggr=false;
> SELECT percentile_approx(cast(substr(src.value,5) AS double), 0.5) FROM src;
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2592) ConstantOI doesn't always propagate to
UDAF
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2592:
------------------------------
Attachment: HIVE-2592.D513.1.patch
jonchang requested code review of "HIVE-2592 [jira] ConstantOI doesn't always propagate to UDAF".
Reviewers: JIRA
constants
Depending on the query plan, a UDAF may not actually get a constant OI. E.g.,
set hive.map.aggr=false;
SELECT percentile_approx(cast(substr(src.value,5) AS double), 0.5) FROM src;
TEST PLAN
EMPTY
REVISION DETAIL
https://reviews.facebook.net/D513
AFFECTED FILES
ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnInfo.java
ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/ExprProcFactory.java
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java
ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeColumnDesc.java
ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeConstantDesc.java
ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDesc.java
ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeFieldDesc.java
ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java
ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeNullDesc.java
ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/1071/
Tip: use the X-Herald-Rules header to filter Herald messages in your client.
> ConstantOI doesn't always propagate to UDAF
> -------------------------------------------
>
> Key: HIVE-2592
> URL: https://issues.apache.org/jira/browse/HIVE-2592
> Project: Hive
> Issue Type: Bug
> Reporter: Jonathan Chang
> Assignee: Jonathan Chang
> Attachments: HIVE-2592.D513.1.patch
>
>
> Depending on the query plan, a UDAF may not actually get a constant OI. E.g.,
> set hive.map.aggr=false;
> SELECT percentile_approx(cast(substr(src.value,5) AS double), 0.5) FROM src;
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2592) ConstantOI doesn't always propagate
to UDAF
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13154759#comment-13154759 ]
Phabricator commented on HIVE-2592:
-----------------------------------
jonchang has commented on the revision "HIVE-2592 [jira] ConstantOI doesn't always propagate to UDAF".
Your wish is my command =). This diff mainly gets most of semanticanalyzer and desc nodes using object inspectors. This makes sure that all paths to plan generation preserve the exact type (and hence constantness).
The next step to everything being copacetic will be updating the serializers.
REVISION DETAIL
https://reviews.facebook.net/D513
> ConstantOI doesn't always propagate to UDAF
> -------------------------------------------
>
> Key: HIVE-2592
> URL: https://issues.apache.org/jira/browse/HIVE-2592
> Project: Hive
> Issue Type: Bug
> Reporter: Jonathan Chang
> Assignee: Jonathan Chang
> Attachments: HIVE-2592.D513.1.patch
>
>
> Depending on the query plan, a UDAF may not actually get a constant OI. E.g.,
> set hive.map.aggr=false;
> SELECT percentile_approx(cast(substr(src.value,5) AS double), 0.5) FROM src;
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2592) ConstantOI doesn't always propagate
to UDAF
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156536#comment-13156536 ]
Phabricator commented on HIVE-2592:
-----------------------------------
ikabiljo has commented on the revision "HIVE-2592 [jira] ConstantOI doesn't always propagate to UDAF".
There must have been a good reason to have TypeInfo/ObjectInspector pair, since it was built. I am wondering if that reason is still valid.
If it is, it might be a best solution to add constant OIs to TypeInfo, and then all different issues around constant OIs would be resolved.
If there is not a good reason for TypeInfo any more, then it might be best to just remove it everywhere (and leave them only for backward compatibility for old UDFs that use them, but create them upon request, without storing them anywhere).
What do you think?
REVISION DETAIL
https://reviews.facebook.net/D513
> ConstantOI doesn't always propagate to UDAF
> -------------------------------------------
>
> Key: HIVE-2592
> URL: https://issues.apache.org/jira/browse/HIVE-2592
> Project: Hive
> Issue Type: Bug
> Reporter: Jonathan Chang
> Assignee: Jonathan Chang
> Attachments: HIVE-2592.D513.1.patch, HIVE-2592.D513.2.patch
>
>
> Depending on the query plan, a UDAF may not actually get a constant OI. E.g.,
> set hive.map.aggr=false;
> SELECT percentile_approx(cast(substr(src.value,5) AS double), 0.5) FROM src;
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2592) ConstantOI doesn't always propagate to
UDAF
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2592:
------------------------------
Attachment: HIVE-2592.D513.2.patch
jonchang updated the revision "HIVE-2592 [jira] ConstantOI doesn't always propagate to UDAF".
Reviewers: JIRA
Rebase ant lint fixes.
REVISION DETAIL
https://reviews.facebook.net/D513
AFFECTED FILES
ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnInfo.java
ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/ExprProcFactory.java
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java
ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeColumnDesc.java
ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeConstantDesc.java
ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDesc.java
ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeFieldDesc.java
ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java
ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeNullDesc.java
ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java
> ConstantOI doesn't always propagate to UDAF
> -------------------------------------------
>
> Key: HIVE-2592
> URL: https://issues.apache.org/jira/browse/HIVE-2592
> Project: Hive
> Issue Type: Bug
> Reporter: Jonathan Chang
> Assignee: Jonathan Chang
> Attachments: HIVE-2592.D513.1.patch, HIVE-2592.D513.2.patch
>
>
> Depending on the query plan, a UDAF may not actually get a constant OI. E.g.,
> set hive.map.aggr=false;
> SELECT percentile_approx(cast(substr(src.value,5) AS double), 0.5) FROM src;
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2592) ConstantOI doesn't always propagate
to UDAF
Posted by "John Sichi (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13154436#comment-13154436 ]
John Sichi commented on HIVE-2592:
----------------------------------
Sure.
> ConstantOI doesn't always propagate to UDAF
> -------------------------------------------
>
> Key: HIVE-2592
> URL: https://issues.apache.org/jira/browse/HIVE-2592
> Project: Hive
> Issue Type: Bug
> Reporter: Jonathan Chang
> Assignee: Jonathan Chang
>
> Depending on the query plan, a UDAF may not actually get a constant OI. E.g.,
> set hive.map.aggr=false;
> SELECT percentile_approx(cast(substr(src.value,5) AS double), 0.5) FROM src;
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2592) ConstantOI doesn't always propagate
to UDAF
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13154955#comment-13154955 ]
Phabricator commented on HIVE-2592:
-----------------------------------
ikabiljo has commented on the revision "HIVE-2592 [jira] ConstantOI doesn't always propagate to UDAF".
I do now understand pretty much anything here :) but just a two questions:
- is there any general usage of TypeInfo, where ObjectInspector cannot be used? (i.e. should TypeInfo be deprecated and only left for backward compatibility for call-sites)
- does this diff/next one tries to fix the issue of try to pass constant OI as partial result to the merge phases?
REVISION DETAIL
https://reviews.facebook.net/D513
> ConstantOI doesn't always propagate to UDAF
> -------------------------------------------
>
> Key: HIVE-2592
> URL: https://issues.apache.org/jira/browse/HIVE-2592
> Project: Hive
> Issue Type: Bug
> Reporter: Jonathan Chang
> Assignee: Jonathan Chang
> Attachments: HIVE-2592.D513.1.patch, HIVE-2592.D513.2.patch
>
>
> Depending on the query plan, a UDAF may not actually get a constant OI. E.g.,
> set hive.map.aggr=false;
> SELECT percentile_approx(cast(substr(src.value,5) AS double), 0.5) FROM src;
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2592) ConstantOI doesn't always propagate
to UDAF
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156537#comment-13156537 ]
Phabricator commented on HIVE-2592:
-----------------------------------
jonchang has commented on the revision "HIVE-2592 [jira] ConstantOI doesn't always propagate to UDAF".
Maybe someone with some more depth on the history here can chime in, but it seems to me that if the TI/OI split was intended to solve the problem, it's a pretty broken solution.
REVISION DETAIL
https://reviews.facebook.net/D513
> ConstantOI doesn't always propagate to UDAF
> -------------------------------------------
>
> Key: HIVE-2592
> URL: https://issues.apache.org/jira/browse/HIVE-2592
> Project: Hive
> Issue Type: Bug
> Reporter: Jonathan Chang
> Assignee: Jonathan Chang
> Attachments: HIVE-2592.D513.1.patch, HIVE-2592.D513.2.patch
>
>
> Depending on the query plan, a UDAF may not actually get a constant OI. E.g.,
> set hive.map.aggr=false;
> SELECT percentile_approx(cast(substr(src.value,5) AS double), 0.5) FROM src;
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2592) ConstantOI doesn't always propagate
to UDAF
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158892#comment-13158892 ]
Phabricator commented on HIVE-2592:
-----------------------------------
jsichi has requested changes to the revision "HIVE-2592 [jira] ConstantOI doesn't always propagate to UDAF".
I got many failures from running ant test on this one. I think at this point it would make sense for you to set up a dev box so you can run through ant test yourself. (Marek has the parallel test stuff ready to try too.)
REVISION DETAIL
https://reviews.facebook.net/D513
> ConstantOI doesn't always propagate to UDAF
> -------------------------------------------
>
> Key: HIVE-2592
> URL: https://issues.apache.org/jira/browse/HIVE-2592
> Project: Hive
> Issue Type: Bug
> Reporter: Jonathan Chang
> Assignee: Jonathan Chang
> Attachments: HIVE-2592.D513.1.patch, HIVE-2592.D513.2.patch
>
>
> Depending on the query plan, a UDAF may not actually get a constant OI. E.g.,
> set hive.map.aggr=false;
> SELECT percentile_approx(cast(substr(src.value,5) AS double), 0.5) FROM src;
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira