You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Jonathan Chang <jo...@fb.com> on 2011/10/03 03:31:21 UTC
Re: Review Request: Improve support for ConstantObjectInspectors in Hive
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2078/
-----------------------------------------------------------
(Updated 2011-10-03 01:31:21.490736)
Review request for hive, Carl Steinbach, John Sichi, and Igor Kabiljo.
Changes
-------
Take suggestions after discussion with ikabiljo and jvs. Get rid of constantfoldablegenericudf in favor of just having both the exprnode evaluator and desc initialize /w constant folding and cache values for all non-stateful, deterministic generic udfs.
Summary
-------
ConstantObjectInspectors were introduced in a previous patch so that UDF's could have access to constant values at initialize time. This diff adds the following improvements:
1.) Extends this functionality beyond primitive types to List/Map types.
2.) Fixes a bug in ExprNodeGenericFuncDesc where the constant value would not propagate beyond one level of evaluation.
3.) Adds a helper class to make functions constant-aware. By inheriting from this class they return constant OIs if all their arguments are constant, and they cache the result so that computation is only done during initialize(). Some functions such as ARRAY/IF/MAP have been migrated.
This addresses bug HIVE-2470.
https://issues.apache.org/jira/browse/HIVE-2470
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java ebc236d
ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java 7da5d6a
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java 570e408
ql/src/test/queries/clientpositive/constant_prop.q PRE-CREATION
ql/src/test/results/clientpositive/constant_prop.q.out PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java a4689fb
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 2c1a7e0
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantListObjectInspector.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantMapObjectInspector.java PRE-CREATION
Diff: https://reviews.apache.org/r/2078/diff
Testing
-------
ant test
Thanks,
Jonathan
Re: Review Request: Improve support for ConstantObjectInspectors in Hive
Posted by Jonathan Chang <jo...@fb.com>.
> On 2011-10-03 08:31:46, Igor Kabiljo wrote:
> > Nice, this is even better then what I thought.
> >
> > This now supports all OIs, except for struct and unions, right? No reason not to add structs, should we add both?
> > Also, should we change so that every VoidObjectInspector is constant (not sure if that would complicate things more, in which case - no)?
> >
> > I still think it would be preferable that constant OI guarantees that objects it returns are constant - if somebody changes returned value, next values it returns shouldn't be affected. But this doesn't matter that much.
> >
* I agree that structs and unions would be good to have at some point but it turns out they are much harder to implement. In particular, they contain references to constituent OIs and do not have natural Java/Writable equivalents. Thus, the way the constant value should be stored is not obvious and you have weird interactions where some field OI maybe a constantOI but not the whole thing, etc. This is complex enough that I think it's worth deferring to another diff
* I will convert void OI to be constant.
* I'm not sure that constant OI guarantee that its objects are constant; it might imply unnecessary copies and currently none of the rest of the UDF architecture functions this way (most things passed around are writable and it's up to consumers not to assume ownership).
- Jonathan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2078/#review2259
-----------------------------------------------------------
On 2011-10-03 01:36:16, Jonathan Chang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/2078/
> -----------------------------------------------------------
>
> (Updated 2011-10-03 01:36:16)
>
>
> Review request for hive, Carl Steinbach, John Sichi, and Igor Kabiljo.
>
>
> Summary
> -------
>
> ConstantObjectInspectors were introduced in a previous patch so that UDF's could have access to constant values at initialize time. This diff adds the following improvements:
>
> 1.) Extends this functionality beyond primitive types to List/Map types.
> 2.) Fixes a bug in ExprNodeGenericFuncDesc where the constant value would not propagate beyond one level of evaluation.
> 3.) Adds a helper class to make functions constant-aware. By inheriting from this class they return constant OIs if all their arguments are constant, and they cache the result so that computation is only done during initialize(). Some functions such as ARRAY/IF/MAP have been migrated.
>
>
> This addresses bug HIVE-2470.
> https://issues.apache.org/jira/browse/HIVE-2470
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java ebc236d
> ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java 7da5d6a
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java 570e408
> ql/src/test/queries/clientpositive/constant_prop.q PRE-CREATION
> ql/src/test/results/clientpositive/constant_prop.q.out PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java a4689fb
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 2c1a7e0
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantListObjectInspector.java PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantMapObjectInspector.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/2078/diff
>
>
> Testing
> -------
>
> ant test
>
>
> Thanks,
>
> Jonathan
>
>
Re: Review Request: Improve support for ConstantObjectInspectors in Hive
Posted by Igor Kabiljo <ik...@fb.com>.
> On 2011-10-03 08:31:46, Igor Kabiljo wrote:
> > Nice, this is even better then what I thought.
> >
> > This now supports all OIs, except for struct and unions, right? No reason not to add structs, should we add both?
> > Also, should we change so that every VoidObjectInspector is constant (not sure if that would complicate things more, in which case - no)?
> >
> > I still think it would be preferable that constant OI guarantees that objects it returns are constant - if somebody changes returned value, next values it returns shouldn't be affected. But this doesn't matter that much.
> >
>
> Jonathan Chang wrote:
> * I agree that structs and unions would be good to have at some point but it turns out they are much harder to implement. In particular, they contain references to constituent OIs and do not have natural Java/Writable equivalents. Thus, the way the constant value should be stored is not obvious and you have weird interactions where some field OI maybe a constantOI but not the whole thing, etc. This is complex enough that I think it's worth deferring to another diff
>
> * I will convert void OI to be constant.
>
> * I'm not sure that constant OI guarantee that its objects are constant; it might imply unnecessary copies and currently none of the rest of the UDF architecture functions this way (most things passed around are writable and it's up to consumers not to assume ownership).
- I didn't know it was like that (for structs and unions), I thought it was straightforward. so definitely for next diff, not this one
- constant OI guarantee will not incur much overhead - since all those are executed only once - since they are constant. but I agree, the rest of architecture functions with that assumption, so it is better to be consistent (and later structs - Object[] - cannot be made constant)
- Igor
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2078/#review2259
-----------------------------------------------------------
On 2011-10-03 01:36:16, Jonathan Chang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/2078/
> -----------------------------------------------------------
>
> (Updated 2011-10-03 01:36:16)
>
>
> Review request for hive, Carl Steinbach, John Sichi, and Igor Kabiljo.
>
>
> Summary
> -------
>
> ConstantObjectInspectors were introduced in a previous patch so that UDF's could have access to constant values at initialize time. This diff adds the following improvements:
>
> 1.) Extends this functionality beyond primitive types to List/Map types.
> 2.) Fixes a bug in ExprNodeGenericFuncDesc where the constant value would not propagate beyond one level of evaluation.
> 3.) Adds a helper class to make functions constant-aware. By inheriting from this class they return constant OIs if all their arguments are constant, and they cache the result so that computation is only done during initialize(). Some functions such as ARRAY/IF/MAP have been migrated.
>
>
> This addresses bug HIVE-2470.
> https://issues.apache.org/jira/browse/HIVE-2470
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java ebc236d
> ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java 7da5d6a
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java 570e408
> ql/src/test/queries/clientpositive/constant_prop.q PRE-CREATION
> ql/src/test/results/clientpositive/constant_prop.q.out PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java a4689fb
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 2c1a7e0
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantListObjectInspector.java PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantMapObjectInspector.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/2078/diff
>
>
> Testing
> -------
>
> ant test
>
>
> Thanks,
>
> Jonathan
>
>
Re: Review Request: Improve support for ConstantObjectInspectors in Hive
Posted by Igor Kabiljo <ik...@fb.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2078/#review2259
-----------------------------------------------------------
Nice, this is even better then what I thought.
This now supports all OIs, except for struct and unions, right? No reason not to add structs, should we add both?
Also, should we change so that every VoidObjectInspector is constant (not sure if that would complicate things more, in which case - no)?
I still think it would be preferable that constant OI guarantees that objects it returns are constant - if somebody changes returned value, next values it returns shouldn't be affected. But this doesn't matter that much.
- Igor
On 2011-10-03 01:36:16, Jonathan Chang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/2078/
> -----------------------------------------------------------
>
> (Updated 2011-10-03 01:36:16)
>
>
> Review request for hive, Carl Steinbach, John Sichi, and Igor Kabiljo.
>
>
> Summary
> -------
>
> ConstantObjectInspectors were introduced in a previous patch so that UDF's could have access to constant values at initialize time. This diff adds the following improvements:
>
> 1.) Extends this functionality beyond primitive types to List/Map types.
> 2.) Fixes a bug in ExprNodeGenericFuncDesc where the constant value would not propagate beyond one level of evaluation.
> 3.) Adds a helper class to make functions constant-aware. By inheriting from this class they return constant OIs if all their arguments are constant, and they cache the result so that computation is only done during initialize(). Some functions such as ARRAY/IF/MAP have been migrated.
>
>
> This addresses bug HIVE-2470.
> https://issues.apache.org/jira/browse/HIVE-2470
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java ebc236d
> ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java 7da5d6a
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java 570e408
> ql/src/test/queries/clientpositive/constant_prop.q PRE-CREATION
> ql/src/test/results/clientpositive/constant_prop.q.out PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java a4689fb
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 2c1a7e0
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantListObjectInspector.java PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantMapObjectInspector.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/2078/diff
>
>
> Testing
> -------
>
> ant test
>
>
> Thanks,
>
> Jonathan
>
>
Re: Review Request: Improve support for ConstantObjectInspectors in Hive
Posted by Jonathan Chang <jo...@fb.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2078/
-----------------------------------------------------------
(Updated 2011-10-17 21:19:30.440767)
Review request for hive, Carl Steinbach, John Sichi, and Igor Kabiljo.
Changes
-------
Rebase!
Summary
-------
ConstantObjectInspectors were introduced in a previous patch so that UDF's could have access to constant values at initialize time. This diff adds the following improvements:
1.) Extends this functionality beyond primitive types to List/Map types.
2.) Fixes a bug in ExprNodeGenericFuncDesc where the constant value would not propagate beyond one level of evaluation.
3.) Adds a helper class to make functions constant-aware. By inheriting from this class they return constant OIs if all their arguments are constant, and they cache the result so that computation is only done during initialize(). Some functions such as ARRAY/IF/MAP have been migrated.
This addresses bug HIVE-2470.
https://issues.apache.org/jira/browse/HIVE-2470
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/SimpleGenericUDAFParameterInfo.java 3277b98
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentileApprox.java 678607b
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/AbstractGenericUDAFResolver.java 766e473
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFParameterInfo.java 04cfc28
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1110cfa
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java cbbfffb
ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndex.java c179d44
Diff: https://reviews.apache.org/r/2078/diff
Testing
-------
ant test
Thanks,
Jonathan
Re: Review Request: Improve support for ConstantObjectInspectors in Hive
Posted by Jonathan Chang <jo...@fb.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2078/
-----------------------------------------------------------
(Updated 2011-10-07 17:36:59.604078)
Review request for hive, Carl Steinbach, John Sichi, and Igor Kabiljo.
Changes
-------
Rebase for realsies.
Summary
-------
ConstantObjectInspectors were introduced in a previous patch so that UDF's could have access to constant values at initialize time. This diff adds the following improvements:
1.) Extends this functionality beyond primitive types to List/Map types.
2.) Fixes a bug in ExprNodeGenericFuncDesc where the constant value would not propagate beyond one level of evaluation.
3.) Adds a helper class to make functions constant-aware. By inheriting from this class they return constant OIs if all their arguments are constant, and they cache the result so that computation is only done during initialize(). Some functions such as ARRAY/IF/MAP have been migrated.
This addresses bug HIVE-2470.
https://issues.apache.org/jira/browse/HIVE-2470
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java ebc236d
ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java 7da5d6a
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java 570e408
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEWAHBitmapEmpty.java aa8d474
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFUtils.java 832f823
ql/src/test/queries/clientpositive/constant_prop.q PRE-CREATION
ql/src/test/results/clientpositive/constant_prop.q.out PRE-CREATION
ql/src/test/results/clientpositive/str_to_map.q.out 30fa7ad
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java a4689fb
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 2d45aba
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantListObjectInspector.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantMapObjectInspector.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 5c8f42d
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantByteObjectInspector.java 5a7fe53
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantTimestampObjectInspector.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantVoidObjectInspector.java f3aee4e
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableVoidObjectInspector.java 6f99561
Diff: https://reviews.apache.org/r/2078/diff
Testing
-------
ant test
Thanks,
Jonathan
Re: Review Request: Improve support for ConstantObjectInspectors in Hive
Posted by Jonathan Chang <jo...@fb.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2078/
-----------------------------------------------------------
(Updated 2011-10-06 21:16:11.713258)
Review request for hive, Carl Steinbach, John Sichi, and Igor Kabiljo.
Changes
-------
Rebased.
Summary
-------
ConstantObjectInspectors were introduced in a previous patch so that UDF's could have access to constant values at initialize time. This diff adds the following improvements:
1.) Extends this functionality beyond primitive types to List/Map types.
2.) Fixes a bug in ExprNodeGenericFuncDesc where the constant value would not propagate beyond one level of evaluation.
3.) Adds a helper class to make functions constant-aware. By inheriting from this class they return constant OIs if all their arguments are constant, and they cache the result so that computation is only done during initialize(). Some functions such as ARRAY/IF/MAP have been migrated.
This addresses bug HIVE-2470.
https://issues.apache.org/jira/browse/HIVE-2470
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java ebc236d
ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java 7da5d6a
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java 570e408
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEWAHBitmapEmpty.java aa8d474
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFUtils.java 832f823
ql/src/test/queries/clientpositive/constant_prop.q PRE-CREATION
ql/src/test/results/clientpositive/constant_prop.q.out PRE-CREATION
ql/src/test/results/clientpositive/str_to_map.q.out 30fa7ad
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java a4689fb
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 2c1a7e0
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantListObjectInspector.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantMapObjectInspector.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java a37547f
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantByteObjectInspector.java 5a7fe53
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantTimestampObjectInspector.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantVoidObjectInspector.java f3aee4e
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableVoidObjectInspector.java 6f99561
Diff: https://reviews.apache.org/r/2078/diff
Testing
-------
ant test
Thanks,
Jonathan
Re: Review Request: Improve support for ConstantObjectInspectors in Hive
Posted by Jonathan Chang <jo...@fb.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2078/
-----------------------------------------------------------
(Updated 2011-10-05 23:32:22.234376)
Review request for hive, Carl Steinbach, John Sichi, and Igor Kabiljo.
Changes
-------
Fixes to make unittests pass:
1.) The return resolver now always converts its input into a standard writable OI so that its output is not a constantoi.
2.) Fix spurious assert on EWAHBitmapEmpty.
3.) Add constant support for TIMESTAMP type.
4.) Make sure key/value/element OIs to ListOI/MapOI are writable.
Summary
-------
ConstantObjectInspectors were introduced in a previous patch so that UDF's could have access to constant values at initialize time. This diff adds the following improvements:
1.) Extends this functionality beyond primitive types to List/Map types.
2.) Fixes a bug in ExprNodeGenericFuncDesc where the constant value would not propagate beyond one level of evaluation.
3.) Adds a helper class to make functions constant-aware. By inheriting from this class they return constant OIs if all their arguments are constant, and they cache the result so that computation is only done during initialize(). Some functions such as ARRAY/IF/MAP have been migrated.
This addresses bug HIVE-2470.
https://issues.apache.org/jira/browse/HIVE-2470
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java 7da5d6a
ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java ebc236d
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java 570e408
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEWAHBitmapEmpty.java aa8d474
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFUtils.java 832f823
ql/src/test/queries/clientpositive/constant_prop.q PRE-CREATION
ql/src/test/results/clientpositive/constant_prop.q.out PRE-CREATION
ql/src/test/results/clientpositive/str_to_map.q.out 30fa7ad
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java a4689fb
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 2c1a7e0
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantListObjectInspector.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantMapObjectInspector.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java a37547f
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantByteObjectInspector.java 5a7fe53
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantTimestampObjectInspector.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantVoidObjectInspector.java f3aee4e
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableVoidObjectInspector.java 6f99561
Diff: https://reviews.apache.org/r/2078/diff
Testing
-------
ant test
Thanks,
Jonathan
Re: Review Request: Improve support for ConstantObjectInspectors in Hive
Posted by Jonathan Chang <jo...@fb.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2078/
-----------------------------------------------------------
(Updated 2011-10-04 17:46:49.734706)
Review request for hive, Carl Steinbach, John Sichi, and Igor Kabiljo.
Changes
-------
Fix a couple of typos.
Summary
-------
ConstantObjectInspectors were introduced in a previous patch so that UDF's could have access to constant values at initialize time. This diff adds the following improvements:
1.) Extends this functionality beyond primitive types to List/Map types.
2.) Fixes a bug in ExprNodeGenericFuncDesc where the constant value would not propagate beyond one level of evaluation.
3.) Adds a helper class to make functions constant-aware. By inheriting from this class they return constant OIs if all their arguments are constant, and they cache the result so that computation is only done during initialize(). Some functions such as ARRAY/IF/MAP have been migrated.
This addresses bug HIVE-2470.
https://issues.apache.org/jira/browse/HIVE-2470
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java ebc236d
ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java 7da5d6a
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java 570e408
ql/src/test/queries/clientpositive/constant_prop.q PRE-CREATION
ql/src/test/results/clientpositive/constant_prop.q.out PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java a4689fb
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 2c1a7e0
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantListObjectInspector.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantMapObjectInspector.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java a37547f
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantVoidObjectInspector.java f3aee4e
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableVoidObjectInspector.java 6f99561
Diff: https://reviews.apache.org/r/2078/diff
Testing
-------
ant test
Thanks,
Jonathan
Re: Review Request: Improve support for ConstantObjectInspectors in Hive
Posted by John Sichi <js...@fb.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2078/#review2291
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
<https://reviews.apache.org/r/2078/#comment5278>
typo: implementation
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
<https://reviews.apache.org/r/2078/#comment5279>
typo: constant
- John
On 2011-10-03 18:35:09, Jonathan Chang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/2078/
> -----------------------------------------------------------
>
> (Updated 2011-10-03 18:35:09)
>
>
> Review request for hive, Carl Steinbach, John Sichi, and Igor Kabiljo.
>
>
> Summary
> -------
>
> ConstantObjectInspectors were introduced in a previous patch so that UDF's could have access to constant values at initialize time. This diff adds the following improvements:
>
> 1.) Extends this functionality beyond primitive types to List/Map types.
> 2.) Fixes a bug in ExprNodeGenericFuncDesc where the constant value would not propagate beyond one level of evaluation.
> 3.) Adds a helper class to make functions constant-aware. By inheriting from this class they return constant OIs if all their arguments are constant, and they cache the result so that computation is only done during initialize(). Some functions such as ARRAY/IF/MAP have been migrated.
>
>
> This addresses bug HIVE-2470.
> https://issues.apache.org/jira/browse/HIVE-2470
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java ebc236d
> ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java 7da5d6a
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java 570e408
> ql/src/test/queries/clientpositive/constant_prop.q PRE-CREATION
> ql/src/test/results/clientpositive/constant_prop.q.out PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java a4689fb
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 2c1a7e0
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantListObjectInspector.java PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantMapObjectInspector.java PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java a37547f
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantVoidObjectInspector.java f3aee4e
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableVoidObjectInspector.java 6f99561
>
> Diff: https://reviews.apache.org/r/2078/diff
>
>
> Testing
> -------
>
> ant test
>
>
> Thanks,
>
> Jonathan
>
>
Re: Review Request: Improve support for ConstantObjectInspectors in Hive
Posted by Jonathan Chang <jo...@fb.com>.
> On 2011-10-03 18:36:46, Igor Kabiljo wrote:
> > looks good!!
Um, @jvs, how do I shipit?
- Jonathan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2078/#review2275
-----------------------------------------------------------
On 2011-10-03 18:35:09, Jonathan Chang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/2078/
> -----------------------------------------------------------
>
> (Updated 2011-10-03 18:35:09)
>
>
> Review request for hive, Carl Steinbach, John Sichi, and Igor Kabiljo.
>
>
> Summary
> -------
>
> ConstantObjectInspectors were introduced in a previous patch so that UDF's could have access to constant values at initialize time. This diff adds the following improvements:
>
> 1.) Extends this functionality beyond primitive types to List/Map types.
> 2.) Fixes a bug in ExprNodeGenericFuncDesc where the constant value would not propagate beyond one level of evaluation.
> 3.) Adds a helper class to make functions constant-aware. By inheriting from this class they return constant OIs if all their arguments are constant, and they cache the result so that computation is only done during initialize(). Some functions such as ARRAY/IF/MAP have been migrated.
>
>
> This addresses bug HIVE-2470.
> https://issues.apache.org/jira/browse/HIVE-2470
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java ebc236d
> ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java 7da5d6a
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java 570e408
> ql/src/test/queries/clientpositive/constant_prop.q PRE-CREATION
> ql/src/test/results/clientpositive/constant_prop.q.out PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java a4689fb
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 2c1a7e0
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantListObjectInspector.java PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantMapObjectInspector.java PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java a37547f
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantVoidObjectInspector.java f3aee4e
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableVoidObjectInspector.java 6f99561
>
> Diff: https://reviews.apache.org/r/2078/diff
>
>
> Testing
> -------
>
> ant test
>
>
> Thanks,
>
> Jonathan
>
>
Re: Review Request: Improve support for ConstantObjectInspectors in Hive
Posted by Igor Kabiljo <ik...@fb.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2078/#review2275
-----------------------------------------------------------
Ship it!
looks good!!
- Igor
On 2011-10-03 18:35:09, Jonathan Chang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/2078/
> -----------------------------------------------------------
>
> (Updated 2011-10-03 18:35:09)
>
>
> Review request for hive, Carl Steinbach, John Sichi, and Igor Kabiljo.
>
>
> Summary
> -------
>
> ConstantObjectInspectors were introduced in a previous patch so that UDF's could have access to constant values at initialize time. This diff adds the following improvements:
>
> 1.) Extends this functionality beyond primitive types to List/Map types.
> 2.) Fixes a bug in ExprNodeGenericFuncDesc where the constant value would not propagate beyond one level of evaluation.
> 3.) Adds a helper class to make functions constant-aware. By inheriting from this class they return constant OIs if all their arguments are constant, and they cache the result so that computation is only done during initialize(). Some functions such as ARRAY/IF/MAP have been migrated.
>
>
> This addresses bug HIVE-2470.
> https://issues.apache.org/jira/browse/HIVE-2470
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java ebc236d
> ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java 7da5d6a
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java 570e408
> ql/src/test/queries/clientpositive/constant_prop.q PRE-CREATION
> ql/src/test/results/clientpositive/constant_prop.q.out PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java a4689fb
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 2c1a7e0
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantListObjectInspector.java PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantMapObjectInspector.java PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java a37547f
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantVoidObjectInspector.java f3aee4e
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableVoidObjectInspector.java 6f99561
>
> Diff: https://reviews.apache.org/r/2078/diff
>
>
> Testing
> -------
>
> ant test
>
>
> Thanks,
>
> Jonathan
>
>
Re: Review Request: Improve support for ConstantObjectInspectors in Hive
Posted by Jonathan Chang <jo...@fb.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2078/
-----------------------------------------------------------
(Updated 2011-10-03 18:35:09.336203)
Review request for hive, Carl Steinbach, John Sichi, and Igor Kabiljo.
Changes
-------
Get rid of writableconstantvoidobjectinspector.
Summary
-------
ConstantObjectInspectors were introduced in a previous patch so that UDF's could have access to constant values at initialize time. This diff adds the following improvements:
1.) Extends this functionality beyond primitive types to List/Map types.
2.) Fixes a bug in ExprNodeGenericFuncDesc where the constant value would not propagate beyond one level of evaluation.
3.) Adds a helper class to make functions constant-aware. By inheriting from this class they return constant OIs if all their arguments are constant, and they cache the result so that computation is only done during initialize(). Some functions such as ARRAY/IF/MAP have been migrated.
This addresses bug HIVE-2470.
https://issues.apache.org/jira/browse/HIVE-2470
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java ebc236d
ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java 7da5d6a
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java 570e408
ql/src/test/queries/clientpositive/constant_prop.q PRE-CREATION
ql/src/test/results/clientpositive/constant_prop.q.out PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java a4689fb
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 2c1a7e0
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantListObjectInspector.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantMapObjectInspector.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java a37547f
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantVoidObjectInspector.java f3aee4e
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableVoidObjectInspector.java 6f99561
Diff: https://reviews.apache.org/r/2078/diff
Testing
-------
ant test
Thanks,
Jonathan
Re: Review Request: Improve support for ConstantObjectInspectors in Hive
Posted by Jonathan Chang <jo...@fb.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2078/
-----------------------------------------------------------
(Updated 2011-10-03 01:36:16.628780)
Review request for hive, Carl Steinbach, John Sichi, and Igor Kabiljo.
Changes
-------
Get rid of some trailing whitespace.
Summary
-------
ConstantObjectInspectors were introduced in a previous patch so that UDF's could have access to constant values at initialize time. This diff adds the following improvements:
1.) Extends this functionality beyond primitive types to List/Map types.
2.) Fixes a bug in ExprNodeGenericFuncDesc where the constant value would not propagate beyond one level of evaluation.
3.) Adds a helper class to make functions constant-aware. By inheriting from this class they return constant OIs if all their arguments are constant, and they cache the result so that computation is only done during initialize(). Some functions such as ARRAY/IF/MAP have been migrated.
This addresses bug HIVE-2470.
https://issues.apache.org/jira/browse/HIVE-2470
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java ebc236d
ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java 7da5d6a
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java 570e408
ql/src/test/queries/clientpositive/constant_prop.q PRE-CREATION
ql/src/test/results/clientpositive/constant_prop.q.out PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java a4689fb
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 2c1a7e0
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantListObjectInspector.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardConstantMapObjectInspector.java PRE-CREATION
Diff: https://reviews.apache.org/r/2078/diff
Testing
-------
ant test
Thanks,
Jonathan