You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Jonathan Chang (Created) (JIRA)" <ji...@apache.org> on 2011/10/16 19:24:11 UTC
[jira] [Created] (HIVE-2506) Get ConstantObjectInspectors working
in UDAFs
Get ConstantObjectInspectors working in UDAFs
---------------------------------------------
Key: HIVE-2506
URL: https://issues.apache.org/jira/browse/HIVE-2506
Project: Hive
Issue Type: Improvement
Reporter: Jonathan Chang
Assignee: Jonathan Chang
ConstantObjectInspectors seem to do the right thing for UDFs, but don't work yet for UDAFs, in part of the complex getEvaluator path which converts back and forth between TypeInfo and ObjectInspectors. We should add an interface so that UDAFs can operate entirely on ObjectInspectors and expose constantness to these UDAFs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2506) Get ConstantObjectInspectors working
in UDAFs
Posted by "Jonathan Chang (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Chang updated HIVE-2506:
---------------------------------
Attachment: HIVE-2506.patch.3
Rebase trunk.
> Get ConstantObjectInspectors working in UDAFs
> ---------------------------------------------
>
> Key: HIVE-2506
> URL: https://issues.apache.org/jira/browse/HIVE-2506
> Project: Hive
> Issue Type: Improvement
> Reporter: Jonathan Chang
> Assignee: Jonathan Chang
> Attachments: HIVE-2506.patch, HIVE-2506.patch.2, HIVE-2506.patch.3
>
>
> ConstantObjectInspectors seem to do the right thing for UDFs, but don't work yet for UDAFs, in part of the complex getEvaluator path which converts back and forth between TypeInfo and ObjectInspectors. We should add an interface so that UDAFs can operate entirely on ObjectInspectors and expose constantness to these UDAFs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-2506) Get ConstantObjectInspectors working
in UDAFs
Posted by "John Sichi (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Sichi resolved HIVE-2506.
------------------------------
Resolution: Fixed
Fix Version/s: 0.9.0
Hadoop Flags: Reviewed
Committed to trunk. Thanks Jonathan!
> Get ConstantObjectInspectors working in UDAFs
> ---------------------------------------------
>
> Key: HIVE-2506
> URL: https://issues.apache.org/jira/browse/HIVE-2506
> Project: Hive
> Issue Type: Improvement
> Reporter: Jonathan Chang
> Assignee: Jonathan Chang
> Fix For: 0.9.0
>
> Attachments: HIVE-2506.patch, HIVE-2506.patch.2, HIVE-2506.patch.3
>
>
> ConstantObjectInspectors seem to do the right thing for UDFs, but don't work yet for UDAFs, in part of the complex getEvaluator path which converts back and forth between TypeInfo and ObjectInspectors. We should add an interface so that UDAFs can operate entirely on ObjectInspectors and expose constantness to these UDAFs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2506) Get ConstantObjectInspectors working
in UDAFs
Posted by "John Sichi (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13129092#comment-13129092 ]
John Sichi commented on HIVE-2506:
----------------------------------
+1. Will commit when tests pass.
> Get ConstantObjectInspectors working in UDAFs
> ---------------------------------------------
>
> Key: HIVE-2506
> URL: https://issues.apache.org/jira/browse/HIVE-2506
> Project: Hive
> Issue Type: Improvement
> Reporter: Jonathan Chang
> Assignee: Jonathan Chang
> Attachments: HIVE-2506.patch, HIVE-2506.patch.2
>
>
> ConstantObjectInspectors seem to do the right thing for UDFs, but don't work yet for UDAFs, in part of the complex getEvaluator path which converts back and forth between TypeInfo and ObjectInspectors. We should add an interface so that UDAFs can operate entirely on ObjectInspectors and expose constantness to these UDAFs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2506) Get ConstantObjectInspectors working
in UDAFs
Posted by "Jonathan Chang (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Chang updated HIVE-2506:
---------------------------------
Attachment: HIVE-2506.patch
> Get ConstantObjectInspectors working in UDAFs
> ---------------------------------------------
>
> Key: HIVE-2506
> URL: https://issues.apache.org/jira/browse/HIVE-2506
> Project: Hive
> Issue Type: Improvement
> Reporter: Jonathan Chang
> Assignee: Jonathan Chang
> Attachments: HIVE-2506.patch
>
>
> ConstantObjectInspectors seem to do the right thing for UDFs, but don't work yet for UDAFs, in part of the complex getEvaluator path which converts back and forth between TypeInfo and ObjectInspectors. We should add an interface so that UDAFs can operate entirely on ObjectInspectors and expose constantness to these UDAFs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2506) Get ConstantObjectInspectors working
in UDAFs
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13128582#comment-13128582 ]
jiraposter@reviews.apache.org commented on HIVE-2506:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2400/
-----------------------------------------------------------
Review request for Carl Steinbach, John Sichi and Igor Kabiljo.
Summary
-------
ConstantOIs are awesome in UDFs, let's also support them in UDAFs. Some side things that needed to happen to make this a reality:
1.) The GenericUDAFParameterInfo was changed to also support ObjectInspectors (and store things in OI rather than TypeInfo format). This ensures that info is most flexibly available to the UDAF, and UDAFs no longer need to understand how to manipulate both TI and OI.
2.) SemanticAnalyzer and FunctionRegistry was changed to use OIs in the node creation process rather than converting back and forth between TI and OI.
3.) AbstractGenericUDAFResolver now overrides the TI method so that subclasses of this don't have to implement both methods.
4.) Changed PercentileApprox as an example of what can be done with the new interface. Things are simplified since the number of buckets and quantiles are known in advance.
This addresses bug HIVE-2506.
https://issues.apache.org/jira/browse/HIVE-2506
Diffs
-----
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java cbbfffb
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1110cfa
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/AbstractGenericUDAFResolver.java 766e473
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFParameterInfo.java 04cfc28
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentileApprox.java 678607b
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/SimpleGenericUDAFParameterInfo.java 3277b98
Diff: https://reviews.apache.org/r/2400/diff
Testing
-------
udaf_percentile_approx.q
Thanks,
Jonathan
> Get ConstantObjectInspectors working in UDAFs
> ---------------------------------------------
>
> Key: HIVE-2506
> URL: https://issues.apache.org/jira/browse/HIVE-2506
> Project: Hive
> Issue Type: Improvement
> Reporter: Jonathan Chang
> Assignee: Jonathan Chang
> Attachments: HIVE-2506.patch
>
>
> ConstantObjectInspectors seem to do the right thing for UDFs, but don't work yet for UDAFs, in part of the complex getEvaluator path which converts back and forth between TypeInfo and ObjectInspectors. We should add an interface so that UDAFs can operate entirely on ObjectInspectors and expose constantness to these UDAFs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2506) Get ConstantObjectInspectors working
in UDAFs
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13129217#comment-13129217 ]
jiraposter@reviews.apache.org commented on HIVE-2506:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2400/
-----------------------------------------------------------
(Updated 2011-10-17 21:21:02.936333)
Review request for Carl Steinbach, John Sichi and Igor Kabiljo.
Changes
-------
Rebase!
Summary
-------
ConstantOIs are awesome in UDFs, let's also support them in UDAFs. Some side things that needed to happen to make this a reality:
1.) The GenericUDAFParameterInfo was changed to also support ObjectInspectors (and store things in OI rather than TypeInfo format). This ensures that info is most flexibly available to the UDAF, and UDAFs no longer need to understand how to manipulate both TI and OI.
2.) SemanticAnalyzer and FunctionRegistry was changed to use OIs in the node creation process rather than converting back and forth between TI and OI.
3.) AbstractGenericUDAFResolver now overrides the TI method so that subclasses of this don't have to implement both methods.
4.) Changed PercentileApprox as an example of what can be done with the new interface. Things are simplified since the number of buckets and quantiles are known in advance.
This addresses bug HIVE-2506.
https://issues.apache.org/jira/browse/HIVE-2506
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/SimpleGenericUDAFParameterInfo.java 3277b98
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/AbstractGenericUDAFResolver.java 766e473
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFParameterInfo.java 04cfc28
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentileApprox.java 678607b
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java cbbfffb
ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndex.java c179d44
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1110cfa
Diff: https://reviews.apache.org/r/2400/diff
Testing
-------
udaf_percentile_approx.q
Thanks,
Jonathan
> Get ConstantObjectInspectors working in UDAFs
> ---------------------------------------------
>
> Key: HIVE-2506
> URL: https://issues.apache.org/jira/browse/HIVE-2506
> Project: Hive
> Issue Type: Improvement
> Reporter: Jonathan Chang
> Assignee: Jonathan Chang
> Attachments: HIVE-2506.patch, HIVE-2506.patch.2, HIVE-2506.patch.3
>
>
> ConstantObjectInspectors seem to do the right thing for UDFs, but don't work yet for UDAFs, in part of the complex getEvaluator path which converts back and forth between TypeInfo and ObjectInspectors. We should add an interface so that UDAFs can operate entirely on ObjectInspectors and expose constantness to these UDAFs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2506) Get ConstantObjectInspectors working
in UDAFs
Posted by "John Sichi (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13129236#comment-13129236 ]
John Sichi commented on HIVE-2506:
----------------------------------
OK trying again with latest.
> Get ConstantObjectInspectors working in UDAFs
> ---------------------------------------------
>
> Key: HIVE-2506
> URL: https://issues.apache.org/jira/browse/HIVE-2506
> Project: Hive
> Issue Type: Improvement
> Reporter: Jonathan Chang
> Assignee: Jonathan Chang
> Attachments: HIVE-2506.patch, HIVE-2506.patch.2, HIVE-2506.patch.3
>
>
> ConstantObjectInspectors seem to do the right thing for UDFs, but don't work yet for UDAFs, in part of the complex getEvaluator path which converts back and forth between TypeInfo and ObjectInspectors. We should add an interface so that UDAFs can operate entirely on ObjectInspectors and expose constantness to these UDAFs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2506) Get ConstantObjectInspectors working
in UDAFs
Posted by "Jonathan Chang (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Chang updated HIVE-2506:
---------------------------------
Attachment: HIVE-2506.patch.2
> Get ConstantObjectInspectors working in UDAFs
> ---------------------------------------------
>
> Key: HIVE-2506
> URL: https://issues.apache.org/jira/browse/HIVE-2506
> Project: Hive
> Issue Type: Improvement
> Reporter: Jonathan Chang
> Assignee: Jonathan Chang
> Attachments: HIVE-2506.patch, HIVE-2506.patch.2
>
>
> ConstantObjectInspectors seem to do the right thing for UDFs, but don't work yet for UDAFs, in part of the complex getEvaluator path which converts back and forth between TypeInfo and ObjectInspectors. We should add an interface so that UDAFs can operate entirely on ObjectInspectors and expose constantness to these UDAFs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2506) Get ConstantObjectInspectors working
in UDAFs
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13128613#comment-13128613 ]
jiraposter@reviews.apache.org commented on HIVE-2506:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2400/
-----------------------------------------------------------
(Updated 2011-10-17 03:03:48.523189)
Review request for Carl Steinbach, John Sichi and Igor Kabiljo.
Changes
-------
Fix an errant comment.
Summary
-------
ConstantOIs are awesome in UDFs, let's also support them in UDAFs. Some side things that needed to happen to make this a reality:
1.) The GenericUDAFParameterInfo was changed to also support ObjectInspectors (and store things in OI rather than TypeInfo format). This ensures that info is most flexibly available to the UDAF, and UDAFs no longer need to understand how to manipulate both TI and OI.
2.) SemanticAnalyzer and FunctionRegistry was changed to use OIs in the node creation process rather than converting back and forth between TI and OI.
3.) AbstractGenericUDAFResolver now overrides the TI method so that subclasses of this don't have to implement both methods.
4.) Changed PercentileApprox as an example of what can be done with the new interface. Things are simplified since the number of buckets and quantiles are known in advance.
This addresses bug HIVE-2506.
https://issues.apache.org/jira/browse/HIVE-2506
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java cbbfffb
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1110cfa
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/AbstractGenericUDAFResolver.java 766e473
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFParameterInfo.java 04cfc28
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentileApprox.java 678607b
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/SimpleGenericUDAFParameterInfo.java 3277b98
Diff: https://reviews.apache.org/r/2400/diff
Testing
-------
udaf_percentile_approx.q
Thanks,
Jonathan
> Get ConstantObjectInspectors working in UDAFs
> ---------------------------------------------
>
> Key: HIVE-2506
> URL: https://issues.apache.org/jira/browse/HIVE-2506
> Project: Hive
> Issue Type: Improvement
> Reporter: Jonathan Chang
> Assignee: Jonathan Chang
> Attachments: HIVE-2506.patch
>
>
> ConstantObjectInspectors seem to do the right thing for UDFs, but don't work yet for UDAFs, in part of the complex getEvaluator path which converts back and forth between TypeInfo and ObjectInspectors. We should add an interface so that UDAFs can operate entirely on ObjectInspectors and expose constantness to these UDAFs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2506) Get ConstantObjectInspectors working
in UDAFs
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13129681#comment-13129681 ]
Hudson commented on HIVE-2506:
------------------------------
Integrated in Hive-trunk-h0.21 #1017 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1017/])
HIVE-2506. Get ConstantObjectInspectors working in UDAFs
(Jonathan Chang via jvs)
jvs : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1185511
Files :
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndex.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/AbstractGenericUDAFResolver.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFParameterInfo.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentileApprox.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/SimpleGenericUDAFParameterInfo.java
> Get ConstantObjectInspectors working in UDAFs
> ---------------------------------------------
>
> Key: HIVE-2506
> URL: https://issues.apache.org/jira/browse/HIVE-2506
> Project: Hive
> Issue Type: Improvement
> Reporter: Jonathan Chang
> Assignee: Jonathan Chang
> Fix For: 0.9.0
>
> Attachments: HIVE-2506.patch, HIVE-2506.patch.2, HIVE-2506.patch.3
>
>
> ConstantObjectInspectors seem to do the right thing for UDFs, but don't work yet for UDAFs, in part of the complex getEvaluator path which converts back and forth between TypeInfo and ObjectInspectors. We should add an interface so that UDAFs can operate entirely on ObjectInspectors and expose constantness to these UDAFs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2506) Get ConstantObjectInspectors working
in UDAFs
Posted by "John Sichi (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13129201#comment-13129201 ]
John Sichi commented on HIVE-2506:
----------------------------------
I got a compilation error, so you probably haven't rebased recently:
{noformat}
[javac] /data/users/jsichi/open/test-trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndex.java:306: getGenericUDAFEvaluator(java.lang.String,java.util.List<org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector>,boolean,boolean) in org.apache.hadoop.hive.ql.exec.FunctionRegistry cannot be applied to (java.lang.String,java.util.List<org.apache.hadoop.hive.serde2.typeinfo.TypeInfo>,boolean,boolean)
[javac] GenericUDAFEvaluator evaluator = FunctionRegistry.getGenericUDAFEvaluator(
{noformat}
> Get ConstantObjectInspectors working in UDAFs
> ---------------------------------------------
>
> Key: HIVE-2506
> URL: https://issues.apache.org/jira/browse/HIVE-2506
> Project: Hive
> Issue Type: Improvement
> Reporter: Jonathan Chang
> Assignee: Jonathan Chang
> Attachments: HIVE-2506.patch, HIVE-2506.patch.2
>
>
> ConstantObjectInspectors seem to do the right thing for UDFs, but don't work yet for UDAFs, in part of the complex getEvaluator path which converts back and forth between TypeInfo and ObjectInspectors. We should add an interface so that UDAFs can operate entirely on ObjectInspectors and expose constantness to these UDAFs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira