You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Jason Dere <jd...@hortonworks.com> on 2014/10/17 04:21:09 UTC
Review Request 26854: HIVE-2573 Create per-session function registry
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/
-----------------------------------------------------------
Review request for hive.
Bugs: HIVE-2573
https://issues.apache.org/jira/browse/HIVE-2573
Repository: hive-git
Description
-------
Small updates to Navis' changes:
- session registry doesn't lookup metastore for UDFs
- my feedback from Navis' original patch
- metastore udfs should not be considered native. This allows them to be added/removed from registry
Diffs
-----
contrib/src/test/results/clientnegative/invalid_row_sequence.q.out 8f3c0b3
contrib/src/test/results/clientpositive/udaf_example_max.q.out 82aeca7
contrib/src/test/results/clientpositive/udaf_example_min.q.out b62ff39
itests/hive-unit/src/test/java/org/apache/hive/service/TestHiveServerSessions.java PRE-CREATION
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java a035ff1
ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 31978fe
ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java 074255b
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java b94f790
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 22e5b47
ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java af633cb
ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
ql/src/test/results/clientpositive/show_functions.q.out 944c567
ql/src/test/results/clientpositive/udaf_collect_set.q.out 42f2d9d
ql/src/test/results/clientpositive/udaf_corr.q.out 1b468c4
ql/src/test/results/clientpositive/udaf_covar_pop.q.out fa1d47e
ql/src/test/results/clientpositive/udaf_covar_samp.q.out ff5b47e
ql/src/test/results/clientpositive/udf_avg.q.out 7ac139f
ql/src/test/results/clientpositive/udf_count.q.out 78228ec
ql/src/test/results/clientpositive/udf_max.q.out 8535752
ql/src/test/results/clientpositive/udf_min.q.out ca8158e
ql/src/test/results/clientpositive/udf_percentile.q.out b963236
ql/src/test/results/clientpositive/udf_std.q.out c004157
ql/src/test/results/clientpositive/udf_stddev.q.out 8948dfc
ql/src/test/results/clientpositive/udf_stddev_samp.q.out 6619ea1
ql/src/test/results/clientpositive/udf_sum.q.out 2f4e3da
ql/src/test/results/clientpositive/udf_var_pop.q.out cd9c61e
ql/src/test/results/clientpositive/udf_var_samp.q.out 190435d
ql/src/test/results/clientpositive/udf_variance.q.out c00dd66
service/src/test/org/apache/hadoop/hive/service/TestHiveServerSessions.java fd38907
Diff: https://reviews.apache.org/r/26854/diff/
Testing
-------
Thanks,
Jason Dere
Re: Review Request 26854: HIVE-2573 Create per-session function
registry
Posted by Thejas Nair <th...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review57443
-----------------------------------------------------------
itests/hive-unit/src/test/java/org/apache/hive/service/TestHiveServerSessions.java
<https://reviews.apache.org/r/26854/#comment98141>
You can do something very similar with jdbc+HS2
see TestJdbcWithSQLAuthorization, TestHS2ImpersonationWithRemoteMS
- Thejas Nair
On Oct. 20, 2014, 6:28 p.m., Jason Dere wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26854/
> -----------------------------------------------------------
>
> (Updated Oct. 20, 2014, 6:28 p.m.)
>
>
> Review request for hive, Navis Ryu and Thejas Nair.
>
>
> Bugs: HIVE-2573
> https://issues.apache.org/jira/browse/HIVE-2573
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Small updates to Navis' changes:
> - session registry doesn't lookup metastore for UDFs
> - my feedback from Navis' original patch
> - metastore udfs should not be considered native. This allows them to be added/removed from registry
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 9ac540e
> ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java 074255b
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java b94f790
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
> ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/translator/SqlFunctionConverter.java 31f906a
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java e43d39f
> ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 22e5b47
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java af633cb
> ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/MatchPath.java aa48a6c
> ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/Noop.java fcf6afd
> ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/NoopStreaming.java 41e0102
> ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/NoopWithMap.java 0b090a9
> ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/NoopWithMapStreaming.java 5d322d3
> ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/WindowingTableFunction.java 903a9b0
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
> ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
> ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out c7405ed
> ql/src/test/results/clientnegative/create_function_nonudf_class.q.out d0dd50a
> ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
> service/src/test/org/apache/hadoop/hive/service/TestHiveServerSessions.java fd38907
>
> Diff: https://reviews.apache.org/r/26854/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jason Dere
>
>
Re: Review Request 26854: HIVE-2573 Create per-session function
registry
Posted by Navis Ryu <na...@nexr.com>.
> On Nov. 14, 2014, 9:24 p.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g, line 1626
> > <https://reviews.apache.org/r/26854/diff/6/?file=762528#file762528line1626>
> >
> > The message here should be "reload function statement"
Fixed
> On Nov. 14, 2014, 9:24 p.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/session/SessionConf.java, line 24
> > <https://reviews.apache.org/r/26854/diff/6/?file=762534#file762534line24>
> >
> > What about the idea of moving static call to resolveFunctions() to SessionState? I thought that would remove the need for SessionConf, because then Hive class would once again be usable during query runtime. Unless you think it's cleaner to use SessionConf to get HiveConf rather than the Hive object.
Done
> On Nov. 14, 2014, 9:24 p.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java, line 105
> > <https://reviews.apache.org/r/26854/diff/6/?file=762515#file762515line105>
> >
> > Ok, so this allows the persistent function list to be reloaded, with an explicit RELOAD command. This should work for now, I suppose it's always possible to add more automatic refreshing of functions later on if folks want that.
Yes, let's do that in following issue.
- Navis
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review61331
-----------------------------------------------------------
On Nov. 13, 2014, 10:15 p.m., Jason Dere wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26854/
> -----------------------------------------------------------
>
> (Updated Nov. 13, 2014, 10:15 p.m.)
>
>
> Review request for hive, Navis Ryu and Thejas Nair.
>
>
> Bugs: HIVE-2573
> https://issues.apache.org/jira/browse/HIVE-2573
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Small updates to Navis' changes:
> - session registry doesn't lookup metastore for UDFs
> - my feedback from Navis' original patch
> - metastore udfs should not be considered native. This allows them to be added/removed from registry
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/common/JavaUtils.java 9aa917c
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java cca57d2
> contrib/src/test/results/clientnegative/invalid_row_sequence.q.out 8f3c0b3
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 88b0791
> ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 292c83c
> ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java 074255b
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 6323387
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
> ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 913288f
> ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 18e40b3
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java b900627
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 13277a9
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 211ab6c
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java e2768ff
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/translator/SqlFunctionConverter.java 7f52c29
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 4b2a81a
> ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 22e5b47
> ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g f412010
> ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g f1365fa
> ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g c960a6b
> ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java 2b239ab
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 6962ee9
> ql/src/java/org/apache/hadoop/hive/ql/plan/FunctionWork.java f968bc1
> ql/src/java/org/apache/hadoop/hive/ql/plan/ReloadFunctionDesc.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionConf.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 2806bd1
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
> ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
> ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out c7405ed
> ql/src/test/results/clientnegative/create_function_nonudf_class.q.out d0dd50a
> ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
> ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out e184787
>
> Diff: https://reviews.apache.org/r/26854/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jason Dere
>
>
Re: Review Request 26854: HIVE-2573 Create per-session function
registry
Posted by Jason Dere <jd...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review61331
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java
<https://reviews.apache.org/r/26854/#comment103167>
Ok, so this allows the persistent function list to be reloaded, with an explicit RELOAD command. This should work for now, I suppose it's always possible to add more automatic refreshing of functions later on if folks want that.
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g
<https://reviews.apache.org/r/26854/#comment102901>
The message here should be "reload function statement"
ql/src/java/org/apache/hadoop/hive/ql/session/SessionConf.java
<https://reviews.apache.org/r/26854/#comment103160>
What about the idea of moving static call to resolveFunctions() to SessionState? I thought that would remove the need for SessionConf, because then Hive class would once again be usable during query runtime. Unless you think it's cleaner to use SessionConf to get HiveConf rather than the Hive object.
- Jason Dere
On Nov. 13, 2014, 10:15 p.m., Jason Dere wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26854/
> -----------------------------------------------------------
>
> (Updated Nov. 13, 2014, 10:15 p.m.)
>
>
> Review request for hive, Navis Ryu and Thejas Nair.
>
>
> Bugs: HIVE-2573
> https://issues.apache.org/jira/browse/HIVE-2573
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Small updates to Navis' changes:
> - session registry doesn't lookup metastore for UDFs
> - my feedback from Navis' original patch
> - metastore udfs should not be considered native. This allows them to be added/removed from registry
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/common/JavaUtils.java 9aa917c
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java cca57d2
> contrib/src/test/results/clientnegative/invalid_row_sequence.q.out 8f3c0b3
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 88b0791
> ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 292c83c
> ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java 074255b
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 6323387
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
> ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 913288f
> ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 18e40b3
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java b900627
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 13277a9
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 211ab6c
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java e2768ff
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/translator/SqlFunctionConverter.java 7f52c29
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 4b2a81a
> ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 22e5b47
> ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g f412010
> ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g f1365fa
> ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g c960a6b
> ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java 2b239ab
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 6962ee9
> ql/src/java/org/apache/hadoop/hive/ql/plan/FunctionWork.java f968bc1
> ql/src/java/org/apache/hadoop/hive/ql/plan/ReloadFunctionDesc.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionConf.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 2806bd1
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
> ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
> ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out c7405ed
> ql/src/test/results/clientnegative/create_function_nonudf_class.q.out d0dd50a
> ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
> ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out e184787
>
> Diff: https://reviews.apache.org/r/26854/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jason Dere
>
>
Re: Review Request 26854: HIVE-2573 Create per-session function
registry
Posted by Jason Dere <jd...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/
-----------------------------------------------------------
(Updated Feb. 12, 2015, 4:25 a.m.)
Review request for hive, Navis Ryu and Thejas Nair.
Changes
-------
Updating with HIVE-2573.15.patch.txt from Navis
Bugs: HIVE-2573
https://issues.apache.org/jira/browse/HIVE-2573
Repository: hive-git
Description
-------
Small updates to Navis' changes:
- session registry doesn't lookup metastore for UDFs
- my feedback from Navis' original patch
- metastore udfs should not be considered native. This allows them to be added/removed from registry
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/common/JavaUtils.java 9aa917c
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 062e520
contrib/src/test/results/clientnegative/invalid_row_sequence.q.out 8f3c0b3
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 5f28d73
ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java a06e858
ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java bd00bd4
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 9a3b8c6
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d72783
ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 3a2a6ee
ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java a29601c
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 4aac39a
ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java f8007e1
ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 69a4545
ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java 2729ceb
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java c8c5369
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 6a319b7
ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 1ef6d1b
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 20c73cd
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 149b788
ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g d37f49f
ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java 2b239ab
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 9397b7d
ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingSpec.java 28afc6b
ql/src/java/org/apache/hadoop/hive/ql/plan/FunctionWork.java f968bc1
ql/src/java/org/apache/hadoop/hive/ql/plan/ReloadFunctionDesc.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java d81b44c
ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/WindowingTableFunction.java 903a9b0
ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out c7405ed
ql/src/test/results/clientnegative/create_function_nonudf_class.q.out d0dd50a
ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out ec981ee
Diff: https://reviews.apache.org/r/26854/diff/
Testing
-------
Thanks,
Jason Dere
Re: Review Request 26854: HIVE-2573 Create per-session function
registry
Posted by Navis Ryu <na...@nexr.com>.
> On Feb. 9, 2015, 9:46 p.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java, line 1970
> > <https://reviews.apache.org/r/26854/diff/7/?file=858817#file858817line1970>
> >
> > Looks like this version of registerTemporaryUDF() is unnecessary, and what caused the stack overflow in TestMacroSemanticAnalyzer during pre-commit tests
Fixed. Thanks.
- Navis
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review71702
-----------------------------------------------------------
On Feb. 9, 2015, 6:56 p.m., Jason Dere wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26854/
> -----------------------------------------------------------
>
> (Updated Feb. 9, 2015, 6:56 p.m.)
>
>
> Review request for hive, Navis Ryu and Thejas Nair.
>
>
> Bugs: HIVE-2573
> https://issues.apache.org/jira/browse/HIVE-2573
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Small updates to Navis' changes:
> - session registry doesn't lookup metastore for UDFs
> - my feedback from Navis' original patch
> - metastore udfs should not be considered native. This allows them to be added/removed from registry
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/common/JavaUtils.java 9aa917c
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8885fd3
> contrib/src/test/results/clientnegative/invalid_row_sequence.q.out 8f3c0b3
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 5f28d73
> ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java a06e858
> ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java bd00bd4
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 29fc99b
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
> ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d72783
> ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 3a2a6ee
> ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java a29601c
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 4aac39a
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java f8007e1
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 69a4545
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java 2729ceb
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java c8c5369
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 6a319b7
> ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 1ef6d1b
> ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 20c73cd
> ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 9c7603c
> ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g bbb8eb3
> ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java 2b239ab
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 9397b7d
> ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingSpec.java 28afc6b
> ql/src/java/org/apache/hadoop/hive/ql/plan/FunctionWork.java f968bc1
> ql/src/java/org/apache/hadoop/hive/ql/plan/ReloadFunctionDesc.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 14bc46a
> ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/WindowingTableFunction.java 903a9b0
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
> ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
> ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out c7405ed
> ql/src/test/results/clientnegative/create_function_nonudf_class.q.out d0dd50a
> ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
> ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out ec981ee
>
> Diff: https://reviews.apache.org/r/26854/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jason Dere
>
>
Re: Review Request 26854: HIVE-2573 Create per-session function
registry
Posted by Jason Dere <jd...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review71702
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
<https://reviews.apache.org/r/26854/#comment117570>
Looks like this version of registerTemporaryUDF() is unnecessary, and what caused the stack overflow in TestMacroSemanticAnalyzer during pre-commit tests
- Jason Dere
On Feb. 9, 2015, 6:56 p.m., Jason Dere wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26854/
> -----------------------------------------------------------
>
> (Updated Feb. 9, 2015, 6:56 p.m.)
>
>
> Review request for hive, Navis Ryu and Thejas Nair.
>
>
> Bugs: HIVE-2573
> https://issues.apache.org/jira/browse/HIVE-2573
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Small updates to Navis' changes:
> - session registry doesn't lookup metastore for UDFs
> - my feedback from Navis' original patch
> - metastore udfs should not be considered native. This allows them to be added/removed from registry
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/common/JavaUtils.java 9aa917c
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8885fd3
> contrib/src/test/results/clientnegative/invalid_row_sequence.q.out 8f3c0b3
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 5f28d73
> ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java a06e858
> ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java bd00bd4
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 29fc99b
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
> ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d72783
> ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 3a2a6ee
> ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java a29601c
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 4aac39a
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java f8007e1
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 69a4545
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java 2729ceb
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java c8c5369
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 6a319b7
> ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 1ef6d1b
> ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 20c73cd
> ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 9c7603c
> ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g bbb8eb3
> ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java 2b239ab
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 9397b7d
> ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingSpec.java 28afc6b
> ql/src/java/org/apache/hadoop/hive/ql/plan/FunctionWork.java f968bc1
> ql/src/java/org/apache/hadoop/hive/ql/plan/ReloadFunctionDesc.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 14bc46a
> ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/WindowingTableFunction.java 903a9b0
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
> ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
> ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out c7405ed
> ql/src/test/results/clientnegative/create_function_nonudf_class.q.out d0dd50a
> ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
> ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out ec981ee
>
> Diff: https://reviews.apache.org/r/26854/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jason Dere
>
>
Re: Review Request 26854: HIVE-2573 Create per-session function
registry
Posted by Navis Ryu <na...@nexr.com>.
> On Feb. 11, 2015, 12:49 a.m., Alexander Pivovarov wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java, line 341
> > <https://reviews.apache.org/r/26854/diff/7/?file=858817#file858817line341>
> >
> > What you think about adding 2 static wrapper methods registerUDF and registerGenericUDF and use old code (without "system." prefix)
Is it necessary? It's clarifying where the function is added to.
- Navis
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review71893
-----------------------------------------------------------
On Feb. 9, 2015, 6:56 p.m., Jason Dere wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26854/
> -----------------------------------------------------------
>
> (Updated Feb. 9, 2015, 6:56 p.m.)
>
>
> Review request for hive, Navis Ryu and Thejas Nair.
>
>
> Bugs: HIVE-2573
> https://issues.apache.org/jira/browse/HIVE-2573
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Small updates to Navis' changes:
> - session registry doesn't lookup metastore for UDFs
> - my feedback from Navis' original patch
> - metastore udfs should not be considered native. This allows them to be added/removed from registry
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/common/JavaUtils.java 9aa917c
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8885fd3
> contrib/src/test/results/clientnegative/invalid_row_sequence.q.out 8f3c0b3
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 5f28d73
> ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java a06e858
> ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java bd00bd4
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 29fc99b
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
> ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d72783
> ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 3a2a6ee
> ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java a29601c
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 4aac39a
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java f8007e1
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 69a4545
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java 2729ceb
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java c8c5369
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 6a319b7
> ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 1ef6d1b
> ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 20c73cd
> ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 9c7603c
> ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g bbb8eb3
> ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java 2b239ab
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 9397b7d
> ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingSpec.java 28afc6b
> ql/src/java/org/apache/hadoop/hive/ql/plan/FunctionWork.java f968bc1
> ql/src/java/org/apache/hadoop/hive/ql/plan/ReloadFunctionDesc.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 14bc46a
> ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/WindowingTableFunction.java 903a9b0
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
> ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
> ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out c7405ed
> ql/src/test/results/clientnegative/create_function_nonudf_class.q.out d0dd50a
> ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
> ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out ec981ee
>
> Diff: https://reviews.apache.org/r/26854/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jason Dere
>
>
Re: Review Request 26854: HIVE-2573 Create per-session function
registry
Posted by Alexander Pivovarov <ap...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review71893
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
<https://reviews.apache.org/r/26854/#comment117811>
What you think about adding 2 static wrapper methods registerUDF and registerGenericUDF and use old code (without "system." prefix)
- Alexander Pivovarov
On Feb. 9, 2015, 6:56 p.m., Jason Dere wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26854/
> -----------------------------------------------------------
>
> (Updated Feb. 9, 2015, 6:56 p.m.)
>
>
> Review request for hive, Navis Ryu and Thejas Nair.
>
>
> Bugs: HIVE-2573
> https://issues.apache.org/jira/browse/HIVE-2573
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Small updates to Navis' changes:
> - session registry doesn't lookup metastore for UDFs
> - my feedback from Navis' original patch
> - metastore udfs should not be considered native. This allows them to be added/removed from registry
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/common/JavaUtils.java 9aa917c
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8885fd3
> contrib/src/test/results/clientnegative/invalid_row_sequence.q.out 8f3c0b3
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 5f28d73
> ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java a06e858
> ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java bd00bd4
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 29fc99b
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
> ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d72783
> ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 3a2a6ee
> ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java a29601c
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 4aac39a
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java f8007e1
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 69a4545
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java 2729ceb
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java c8c5369
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 6a319b7
> ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 1ef6d1b
> ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 20c73cd
> ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 9c7603c
> ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g bbb8eb3
> ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java 2b239ab
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 9397b7d
> ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingSpec.java 28afc6b
> ql/src/java/org/apache/hadoop/hive/ql/plan/FunctionWork.java f968bc1
> ql/src/java/org/apache/hadoop/hive/ql/plan/ReloadFunctionDesc.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 14bc46a
> ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/WindowingTableFunction.java 903a9b0
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
> ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
> ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out c7405ed
> ql/src/test/results/clientnegative/create_function_nonudf_class.q.out d0dd50a
> ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
> ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out ec981ee
>
> Diff: https://reviews.apache.org/r/26854/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jason Dere
>
>
Re: Review Request 26854: HIVE-2573 Create per-session function
registry
Posted by Navis Ryu <na...@nexr.com>.
> On Feb. 11, 2015, 12:39 a.m., Alexander Pivovarov wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java, line 265
> > <https://reviews.apache.org/r/26854/diff/7/?file=858816#file858816line265>
> >
> > remove spaces pls
done
> On Feb. 11, 2015, 12:39 a.m., Alexander Pivovarov wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java, line 427
> > <https://reviews.apache.org/r/26854/diff/7/?file=858817#file858817line427>
> >
> > new udf next_day GenericUDFNextDay was added yesterday. Probably should be added here as well
done
- Navis
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review71887
-----------------------------------------------------------
On Feb. 9, 2015, 6:56 p.m., Jason Dere wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26854/
> -----------------------------------------------------------
>
> (Updated Feb. 9, 2015, 6:56 p.m.)
>
>
> Review request for hive, Navis Ryu and Thejas Nair.
>
>
> Bugs: HIVE-2573
> https://issues.apache.org/jira/browse/HIVE-2573
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Small updates to Navis' changes:
> - session registry doesn't lookup metastore for UDFs
> - my feedback from Navis' original patch
> - metastore udfs should not be considered native. This allows them to be added/removed from registry
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/common/JavaUtils.java 9aa917c
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8885fd3
> contrib/src/test/results/clientnegative/invalid_row_sequence.q.out 8f3c0b3
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 5f28d73
> ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java a06e858
> ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java bd00bd4
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 29fc99b
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
> ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d72783
> ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 3a2a6ee
> ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java a29601c
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 4aac39a
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java f8007e1
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 69a4545
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java 2729ceb
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java c8c5369
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 6a319b7
> ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 1ef6d1b
> ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 20c73cd
> ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 9c7603c
> ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g bbb8eb3
> ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java 2b239ab
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 9397b7d
> ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingSpec.java 28afc6b
> ql/src/java/org/apache/hadoop/hive/ql/plan/FunctionWork.java f968bc1
> ql/src/java/org/apache/hadoop/hive/ql/plan/ReloadFunctionDesc.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 14bc46a
> ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/WindowingTableFunction.java 903a9b0
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
> ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
> ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out c7405ed
> ql/src/test/results/clientnegative/create_function_nonudf_class.q.out d0dd50a
> ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
> ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out ec981ee
>
> Diff: https://reviews.apache.org/r/26854/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jason Dere
>
>
Re: Review Request 26854: HIVE-2573 Create per-session function
registry
Posted by Alexander Pivovarov <ap...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review71887
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java
<https://reviews.apache.org/r/26854/#comment117794>
remove spaces pls
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
<https://reviews.apache.org/r/26854/#comment117799>
new udf next_day GenericUDFNextDay was added yesterday. Probably should be added here as well
- Alexander Pivovarov
On Feb. 9, 2015, 6:56 p.m., Jason Dere wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26854/
> -----------------------------------------------------------
>
> (Updated Feb. 9, 2015, 6:56 p.m.)
>
>
> Review request for hive, Navis Ryu and Thejas Nair.
>
>
> Bugs: HIVE-2573
> https://issues.apache.org/jira/browse/HIVE-2573
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Small updates to Navis' changes:
> - session registry doesn't lookup metastore for UDFs
> - my feedback from Navis' original patch
> - metastore udfs should not be considered native. This allows them to be added/removed from registry
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/common/JavaUtils.java 9aa917c
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8885fd3
> contrib/src/test/results/clientnegative/invalid_row_sequence.q.out 8f3c0b3
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 5f28d73
> ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java a06e858
> ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java bd00bd4
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 29fc99b
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
> ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d72783
> ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 3a2a6ee
> ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java a29601c
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 4aac39a
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java f8007e1
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 69a4545
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java 2729ceb
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java c8c5369
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 6a319b7
> ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 1ef6d1b
> ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 20c73cd
> ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 9c7603c
> ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g bbb8eb3
> ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java 2b239ab
> ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 9397b7d
> ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingSpec.java 28afc6b
> ql/src/java/org/apache/hadoop/hive/ql/plan/FunctionWork.java f968bc1
> ql/src/java/org/apache/hadoop/hive/ql/plan/ReloadFunctionDesc.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 14bc46a
> ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/WindowingTableFunction.java 903a9b0
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
> ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
> ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out c7405ed
> ql/src/test/results/clientnegative/create_function_nonudf_class.q.out d0dd50a
> ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
> ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out ec981ee
>
> Diff: https://reviews.apache.org/r/26854/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jason Dere
>
>
Re: Review Request 26854: HIVE-2573 Create per-session function
registry
Posted by Jason Dere <jd...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/
-----------------------------------------------------------
(Updated Feb. 9, 2015, 6:56 p.m.)
Review request for hive, Navis Ryu and Thejas Nair.
Changes
-------
Updating with HIVE-2573.14.patch.txt from Navis
Bugs: HIVE-2573
https://issues.apache.org/jira/browse/HIVE-2573
Repository: hive-git
Description
-------
Small updates to Navis' changes:
- session registry doesn't lookup metastore for UDFs
- my feedback from Navis' original patch
- metastore udfs should not be considered native. This allows them to be added/removed from registry
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/common/JavaUtils.java 9aa917c
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8885fd3
contrib/src/test/results/clientnegative/invalid_row_sequence.q.out 8f3c0b3
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 5f28d73
ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java a06e858
ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java bd00bd4
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 29fc99b
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d72783
ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 3a2a6ee
ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java a29601c
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 4aac39a
ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java f8007e1
ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 69a4545
ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java 2729ceb
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java c8c5369
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 6a319b7
ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 1ef6d1b
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 20c73cd
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 9c7603c
ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g bbb8eb3
ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java 2b239ab
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 9397b7d
ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingSpec.java 28afc6b
ql/src/java/org/apache/hadoop/hive/ql/plan/FunctionWork.java f968bc1
ql/src/java/org/apache/hadoop/hive/ql/plan/ReloadFunctionDesc.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 14bc46a
ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/WindowingTableFunction.java 903a9b0
ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out c7405ed
ql/src/test/results/clientnegative/create_function_nonudf_class.q.out d0dd50a
ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out ec981ee
Diff: https://reviews.apache.org/r/26854/diff/
Testing
-------
Thanks,
Jason Dere
Re: Review Request 26854: HIVE-2573 Create per-session function
registry
Posted by Jason Dere <jd...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/
-----------------------------------------------------------
(Updated Nov. 13, 2014, 10:15 p.m.)
Review request for hive, Navis Ryu and Thejas Nair.
Changes
-------
Updating with patch HIVE-2573.13.patch.txt from Navis
Bugs: HIVE-2573
https://issues.apache.org/jira/browse/HIVE-2573
Repository: hive-git
Description
-------
Small updates to Navis' changes:
- session registry doesn't lookup metastore for UDFs
- my feedback from Navis' original patch
- metastore udfs should not be considered native. This allows them to be added/removed from registry
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/common/JavaUtils.java 9aa917c
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java cca57d2
contrib/src/test/results/clientnegative/invalid_row_sequence.q.out 8f3c0b3
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 88b0791
ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 292c83c
ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java 074255b
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 6323387
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 913288f
ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 18e40b3
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java b900627
ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 13277a9
ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 211ab6c
ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java e2768ff
ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/translator/SqlFunctionConverter.java 7f52c29
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 4b2a81a
ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 22e5b47
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g f412010
ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g f1365fa
ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g c960a6b
ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java 2b239ab
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 6962ee9
ql/src/java/org/apache/hadoop/hive/ql/plan/FunctionWork.java f968bc1
ql/src/java/org/apache/hadoop/hive/ql/plan/ReloadFunctionDesc.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/session/SessionConf.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 2806bd1
ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out c7405ed
ql/src/test/results/clientnegative/create_function_nonudf_class.q.out d0dd50a
ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out e184787
Diff: https://reviews.apache.org/r/26854/diff/
Testing
-------
Thanks,
Jason Dere
Re: Review Request 26854: HIVE-2573 Create per-session function
registry
Posted by Navis Ryu <na...@nexr.com>.
> On Oct. 31, 2014, 1:32 a.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java, line 1594
> > <https://reviews.apache.org/r/26854/diff/3-5/?file=729496#file729496line1594>
> >
> > Are these changes meant for this Jira?
> >
> > Hmm, I think I see why - have the persistent function lookups done during static initialization now caused the Hive class to not be instantiable during runtime? Could you try moving the persistent function lookups out of static initialization, and into a method, which gets called (but only initialized once) during SessionState.start()? Would that take care of the issue?
Hive.class is an access point to metastore, which should not be referenced from runtime classes like Table, Task, etc. Theses changes are for that. Decoupling persistent function registration code parts from static initializer seemed good idea. I'll try that.
> On Oct. 31, 2014, 1:32 a.m., Jason Dere wrote:
> > service/src/test/org/apache/hadoop/hive/service/TestHiveServerSessions.java, line 134
> > <https://reviews.apache.org/r/26854/diff/3-5/?file=729506#file729506line134>
> >
> > What's the issue here?
I tried but couldn't make run MR in hadoop-2. Any idea?
> On Oct. 31, 2014, 1:32 a.m., Jason Dere wrote:
> > itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithMiniKdc.java, line 176
> > <https://reviews.apache.org/r/26854/diff/5/?file=745070#file745070line176>
> >
> > Is this change relevant to this Jira? Or is this just a general fix to TestJdbcWithMiniKdc.testNegativeTokenAuth, which I have noticed to be failing consistently in the precommit tests?
This is my bad from HIVE-8186. Chainging error message will make this test pass, but HIVE-8481 would be better place to to. I'll removed this part.
> On Oct. 31, 2014, 1:32 a.m., Jason Dere wrote:
> > metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java, line 144
> > <https://reviews.apache.org/r/26854/diff/5/?file=745071#file745071line144>
> >
> > Are the changes in this file meant for this Jira?
This is called from static initializer in Hive(class). ObjectStore always returns collection but it's not true in this class. Can be fixed by checking null but seemed better to fix this for consistency.
> On Oct. 31, 2014, 1:32 a.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java, line 399
> > <https://reviews.apache.org/r/26854/diff/5/?file=745080#file745080line399>
> >
> > Was this line meant to be in here?
Yes, my bad. The remnant of futile attempts to run TestHiveServerSessions in hadoop-2. Will be removed.
- Navis
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review59281
-----------------------------------------------------------
On Oct. 30, 2014, 11:41 p.m., Jason Dere wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26854/
> -----------------------------------------------------------
>
> (Updated Oct. 30, 2014, 11:41 p.m.)
>
>
> Review request for hive, Navis Ryu and Thejas Nair.
>
>
> Bugs: HIVE-2573
> https://issues.apache.org/jira/browse/HIVE-2573
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Small updates to Navis' changes:
> - session registry doesn't lookup metastore for UDFs
> - my feedback from Navis' original patch
> - metastore udfs should not be considered native. This allows them to be added/removed from registry
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/common/JavaUtils.java 9aa917c
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7d8e5bc
> contrib/src/test/results/clientnegative/invalid_row_sequence.q.out 8f3c0b3
> itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithMiniKdc.java 6647ce5
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 88b0791
> ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 9ac540e
> ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java 074255b
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java e43a328
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
> ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 5bdeb92
> ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 4e3df75
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java b900627
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 13277a9
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 211ab6c
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java e2768ff
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/translator/SqlFunctionConverter.java 793f117
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 1796b7b
> ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 22e5b47
> ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java 2b239ab
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionConf.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java af633cb
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
> ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
> ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out c7405ed
> ql/src/test/results/clientnegative/create_function_nonudf_class.q.out d0dd50a
> ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
> ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out e184787
> service/src/test/org/apache/hadoop/hive/service/TestHiveServerSessions.java fd38907
>
> Diff: https://reviews.apache.org/r/26854/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jason Dere
>
>
Re: Review Request 26854: HIVE-2573 Create per-session function
registry
Posted by Jason Dere <jd...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review59281
-----------------------------------------------------------
Still have the concerns from one of the previous review comments that there is no longer a way to query for persistent UDFs after static intitialization, since this can potentially cause issues if one or more clients is using Hive CLI.
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
<https://reviews.apache.org/r/26854/#comment100557>
Are these changes meant for this Jira?
Hmm, I think I see why - have the persistent function lookups done during static initialization now caused the Hive class to not be instantiable during runtime? Could you try moving the persistent function lookups out of static initialization, and into a method, which gets called (but only initialized once) during SessionState.start()? Would that take care of the issue?
service/src/test/org/apache/hadoop/hive/service/TestHiveServerSessions.java
<https://reviews.apache.org/r/26854/#comment100560>
What's the issue here?
itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithMiniKdc.java
<https://reviews.apache.org/r/26854/#comment100554>
Is this change relevant to this Jira? Or is this just a general fix to TestJdbcWithMiniKdc.testNegativeTokenAuth, which I have noticed to be failing consistently in the precommit tests?
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
<https://reviews.apache.org/r/26854/#comment100555>
Are the changes in this file meant for this Jira?
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java
<https://reviews.apache.org/r/26854/#comment100544>
Was this line meant to be in here?
- Jason Dere
On Oct. 30, 2014, 11:41 p.m., Jason Dere wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26854/
> -----------------------------------------------------------
>
> (Updated Oct. 30, 2014, 11:41 p.m.)
>
>
> Review request for hive, Navis Ryu and Thejas Nair.
>
>
> Bugs: HIVE-2573
> https://issues.apache.org/jira/browse/HIVE-2573
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Small updates to Navis' changes:
> - session registry doesn't lookup metastore for UDFs
> - my feedback from Navis' original patch
> - metastore udfs should not be considered native. This allows them to be added/removed from registry
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/common/JavaUtils.java 9aa917c
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7d8e5bc
> contrib/src/test/results/clientnegative/invalid_row_sequence.q.out 8f3c0b3
> itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithMiniKdc.java 6647ce5
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 88b0791
> ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 9ac540e
> ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java 074255b
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java e43a328
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
> ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 5bdeb92
> ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 4e3df75
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java b900627
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 13277a9
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 211ab6c
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java e2768ff
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/translator/SqlFunctionConverter.java 793f117
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 1796b7b
> ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 22e5b47
> ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java 2b239ab
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionConf.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java af633cb
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
> ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
> ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out c7405ed
> ql/src/test/results/clientnegative/create_function_nonudf_class.q.out d0dd50a
> ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
> ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out e184787
> service/src/test/org/apache/hadoop/hive/service/TestHiveServerSessions.java fd38907
>
> Diff: https://reviews.apache.org/r/26854/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jason Dere
>
>
Re: Review Request 26854: HIVE-2573 Create per-session function
registry
Posted by Jason Dere <jd...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/
-----------------------------------------------------------
(Updated Oct. 30, 2014, 11:41 p.m.)
Review request for hive, Navis Ryu and Thejas Nair.
Changes
-------
Updating with HIVE-2573.11.patch.txt from Navis
Bugs: HIVE-2573
https://issues.apache.org/jira/browse/HIVE-2573
Repository: hive-git
Description
-------
Small updates to Navis' changes:
- session registry doesn't lookup metastore for UDFs
- my feedback from Navis' original patch
- metastore udfs should not be considered native. This allows them to be added/removed from registry
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/common/JavaUtils.java 9aa917c
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7d8e5bc
contrib/src/test/results/clientnegative/invalid_row_sequence.q.out 8f3c0b3
itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithMiniKdc.java 6647ce5
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 88b0791
ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 9ac540e
ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java 074255b
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java e43a328
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 5bdeb92
ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 4e3df75
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java b900627
ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 13277a9
ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 211ab6c
ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java e2768ff
ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/translator/SqlFunctionConverter.java 793f117
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 1796b7b
ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 22e5b47
ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java 2b239ab
ql/src/java/org/apache/hadoop/hive/ql/session/SessionConf.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java af633cb
ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out c7405ed
ql/src/test/results/clientnegative/create_function_nonudf_class.q.out d0dd50a
ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out e184787
service/src/test/org/apache/hadoop/hive/service/TestHiveServerSessions.java fd38907
Diff: https://reviews.apache.org/r/26854/diff/
Testing
-------
Thanks,
Jason Dere
Re: Review Request 26854: HIVE-2573 Create per-session function
registry
Posted by Jason Dere <jd...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/
-----------------------------------------------------------
(Updated Oct. 24, 2014, 5:34 p.m.)
Review request for hive, Navis Ryu and Thejas Nair.
Changes
-------
Updating with HIVE-2573.10.patch.txt from Navis
Bugs: HIVE-2573
https://issues.apache.org/jira/browse/HIVE-2573
Repository: hive-git
Description
-------
Small updates to Navis' changes:
- session registry doesn't lookup metastore for UDFs
- my feedback from Navis' original patch
- metastore udfs should not be considered native. This allows them to be added/removed from registry
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/common/JavaUtils.java 9aa917c
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 88b0791
ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 9ac540e
ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java 074255b
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java e43a328
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7443f8a
ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java b900627
ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 13277a9
ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 211ab6c
ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java e2768ff
ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/translator/SqlFunctionConverter.java 793f117
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 1796b7b
ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 22e5b47
ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java 2b239ab
ql/src/java/org/apache/hadoop/hive/ql/session/SessionConf.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java af633cb
ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out c7405ed
ql/src/test/results/clientnegative/create_function_nonudf_class.q.out d0dd50a
ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
service/src/test/org/apache/hadoop/hive/service/TestHiveServerSessions.java fd38907
Diff: https://reviews.apache.org/r/26854/diff/
Testing
-------
Thanks,
Jason Dere
Re: Review Request 26854: HIVE-2573 Create per-session function
registry
Posted by Navis Ryu <na...@nexr.com>.
> On Oct. 23, 2014, 9:50 p.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java, line 42
> > <https://reviews.apache.org/r/26854/diff/1-3/?file=723906#file723906line42>
> >
> > Can we replace isNative/isPersistent with an enum that has BUILTIN, PERMANENT, TEMPORARY (or equivalent terms)?
Sure.
> On Oct. 23, 2014, 9:50 p.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java, line 128
> > <https://reviews.apache.org/r/26854/diff/1-3/?file=723908#file723908line128>
> >
> > Why was this check removed? If you are permanent UDFs with Hive CLI, you would have to make sure the UDF resources are available from the cluster as opposed to just on the local filesystem of the client that created the UDF.
Seemed removed by mistake. I'll revert that.
> On Oct. 23, 2014, 9:50 p.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java, line 433
> > <https://reviews.apache.org/r/26854/diff/1-3/?file=723909#file723909line433>
> >
> > I thought builtin functions aren't allowed to be removed?
> > Does this mean that we could create a function using the same class as a built-in function (create a synonym), and deleting this new function will cause this class to be removed from the builtin set?
This should be removed. Thanks.
> On Oct. 23, 2014, 9:50 p.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java, line 465
> > <https://reviews.apache.org/r/26854/diff/1-3/?file=723909#file723909line465>
> >
> > There is no longer a way to query the metastore for UDFs apart from the static initialization. So if one CLI user creates a permanent UDF, another user on CLI, or HS2, will not be able to use that new UDF if the 2nd CLI or HS2 was initialized before this UDF was created.
Permanent functions (persistent function seemed better name, imho) are registered to system registry, which is shared to all clients. So if one user creates new permanent function, it's shared to all clients. The time a user accesses the function, the class is loaded with required resources and registered to session registry as a temporary function.
> On Oct. 23, 2014, 9:50 p.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java, line 511
> > <https://reviews.apache.org/r/26854/diff/1-3/?file=723909#file723909line511>
> >
> > I think I see what you're trying do here, trying to add a mechanism so that if a function is deleted in one session, the other sessions will also see it as discarded if they try to look it up. But I don't actually see discarded being set to true.
My mistake. Fixed.
> On Oct. 23, 2014, 9:50 p.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java, line 152
> > <https://reviews.apache.org/r/26854/diff/3/?file=729496#file729496line152>
> >
> > I don't think it's necessary to pre-emptively query the metastore for permanent UDFs during initialization. If we have user on Hive CLI, we will automatically lookup metastore/download UDF resources, when they may not even be using any of these UDFs during their session. How about we keep the existing behavior that we only look them up when they are used during a query?
> >
> > Also, if we are doing this during static initialization, is Hive be in a state that it can query the metastore? Not sure if there is any other initialization that may need to take place beforehand.
bq. we only look them up when they are used during a query?
This method just stores meta information (classname, resources, etc.) for the function without loading any resources/classes, lessening redundant metastore accesses from all clients. When user accesses the function it's registered to session registry with the information as described above.
bq. Not sure if there is any other initialization that may need to take place beforehand.
Afaik, metastore has static lock for initialization which is checked before client accesses internal of it. If it does not act like that, we should fix metastore.
- Navis
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review57952
-----------------------------------------------------------
On Oct. 23, 2014, 12:20 a.m., Jason Dere wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26854/
> -----------------------------------------------------------
>
> (Updated Oct. 23, 2014, 12:20 a.m.)
>
>
> Review request for hive, Navis Ryu and Thejas Nair.
>
>
> Bugs: HIVE-2573
> https://issues.apache.org/jira/browse/HIVE-2573
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Small updates to Navis' changes:
> - session registry doesn't lookup metastore for UDFs
> - my feedback from Navis' original patch
> - metastore udfs should not be considered native. This allows them to be added/removed from registry
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 9ac540e
> ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java 074255b
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 08e1136
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
> ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java b900627
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/translator/SqlFunctionConverter.java 31f906a
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java e43d39f
> ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 22e5b47
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java af633cb
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
> ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
> ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out c7405ed
> ql/src/test/results/clientnegative/create_function_nonudf_class.q.out d0dd50a
> ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
> service/src/test/org/apache/hadoop/hive/service/TestHiveServerSessions.java fd38907
>
> Diff: https://reviews.apache.org/r/26854/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jason Dere
>
>
Re: Review Request 26854: HIVE-2573 Create per-session function
registry
Posted by Jason Dere <jd...@hortonworks.com>.
> On Oct. 23, 2014, 9:50 p.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java, line 465
> > <https://reviews.apache.org/r/26854/diff/1-3/?file=723909#file723909line465>
> >
> > There is no longer a way to query the metastore for UDFs apart from the static initialization. So if one CLI user creates a permanent UDF, another user on CLI, or HS2, will not be able to use that new UDF if the 2nd CLI or HS2 was initialized before this UDF was created.
>
> Navis Ryu wrote:
> Permanent functions (persistent function seemed better name, imho) are registered to system registry, which is shared to all clients. So if one user creates new permanent function, it's shared to all clients. The time a user accesses the function, the class is loaded with required resources and registered to session registry as a temporary function.
>
> Jason Dere wrote:
> So this would work if all clients are using hiveserver2, because all clients in this scenario would share the same system registry.
> But if one or more clients are using the Hive CLI, any persistent UDFs created/dropped by this CLI client would not be reflected in the other clients (or HS2), since it's a different process/system registry.
>
> Navis Ryu wrote:
> I've missed this message. Yes, it will act like you've commented. But is it a common case to expect things done in separate VM refleceted to others? Should dropping presistent function by a jdbc client be reflected to other CLI clients?
Would be good to have if it's possible. Certainly if one client creates a table, we would want the other client to be able to see it. Being able to have the same behavior with functions seems consistent. I suppose it is a bit harder for this to work in the case of drop function, unless we are looking up the UDF in the metastore each time it is used in a query.
- Jason
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review57952
-----------------------------------------------------------
On Oct. 30, 2014, 11:41 p.m., Jason Dere wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26854/
> -----------------------------------------------------------
>
> (Updated Oct. 30, 2014, 11:41 p.m.)
>
>
> Review request for hive, Navis Ryu and Thejas Nair.
>
>
> Bugs: HIVE-2573
> https://issues.apache.org/jira/browse/HIVE-2573
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Small updates to Navis' changes:
> - session registry doesn't lookup metastore for UDFs
> - my feedback from Navis' original patch
> - metastore udfs should not be considered native. This allows them to be added/removed from registry
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/common/JavaUtils.java 9aa917c
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7d8e5bc
> contrib/src/test/results/clientnegative/invalid_row_sequence.q.out 8f3c0b3
> itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithMiniKdc.java 6647ce5
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 88b0791
> ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 9ac540e
> ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java 074255b
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java e43a328
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
> ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 5bdeb92
> ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 4e3df75
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java b900627
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 13277a9
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 211ab6c
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java e2768ff
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/translator/SqlFunctionConverter.java 793f117
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 1796b7b
> ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 22e5b47
> ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java 2b239ab
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionConf.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java af633cb
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
> ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
> ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out c7405ed
> ql/src/test/results/clientnegative/create_function_nonudf_class.q.out d0dd50a
> ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
> ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out e184787
> service/src/test/org/apache/hadoop/hive/service/TestHiveServerSessions.java fd38907
>
> Diff: https://reviews.apache.org/r/26854/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jason Dere
>
>
Re: Review Request 26854: HIVE-2573 Create per-session function
registry
Posted by Jason Dere <jd...@hortonworks.com>.
> On Oct. 23, 2014, 9:50 p.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java, line 465
> > <https://reviews.apache.org/r/26854/diff/1-3/?file=723909#file723909line465>
> >
> > There is no longer a way to query the metastore for UDFs apart from the static initialization. So if one CLI user creates a permanent UDF, another user on CLI, or HS2, will not be able to use that new UDF if the 2nd CLI or HS2 was initialized before this UDF was created.
>
> Navis Ryu wrote:
> Permanent functions (persistent function seemed better name, imho) are registered to system registry, which is shared to all clients. So if one user creates new permanent function, it's shared to all clients. The time a user accesses the function, the class is loaded with required resources and registered to session registry as a temporary function.
So this would work if all clients are using hiveserver2, because all clients in this scenario would share the same system registry.
But if one or more clients are using the Hive CLI, any persistent UDFs created/dropped by this CLI client would not be reflected in the other clients (or HS2), since it's a different process/system registry.
- Jason
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review57952
-----------------------------------------------------------
On Oct. 23, 2014, 12:20 a.m., Jason Dere wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26854/
> -----------------------------------------------------------
>
> (Updated Oct. 23, 2014, 12:20 a.m.)
>
>
> Review request for hive, Navis Ryu and Thejas Nair.
>
>
> Bugs: HIVE-2573
> https://issues.apache.org/jira/browse/HIVE-2573
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Small updates to Navis' changes:
> - session registry doesn't lookup metastore for UDFs
> - my feedback from Navis' original patch
> - metastore udfs should not be considered native. This allows them to be added/removed from registry
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 9ac540e
> ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java 074255b
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 08e1136
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
> ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java b900627
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/translator/SqlFunctionConverter.java 31f906a
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java e43d39f
> ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 22e5b47
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java af633cb
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
> ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
> ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out c7405ed
> ql/src/test/results/clientnegative/create_function_nonudf_class.q.out d0dd50a
> ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
> service/src/test/org/apache/hadoop/hive/service/TestHiveServerSessions.java fd38907
>
> Diff: https://reviews.apache.org/r/26854/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jason Dere
>
>
Re: Review Request 26854: HIVE-2573 Create per-session function
registry
Posted by Navis Ryu <na...@nexr.com>.
> On Oct. 23, 2014, 9:50 p.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java, line 465
> > <https://reviews.apache.org/r/26854/diff/1-3/?file=723909#file723909line465>
> >
> > There is no longer a way to query the metastore for UDFs apart from the static initialization. So if one CLI user creates a permanent UDF, another user on CLI, or HS2, will not be able to use that new UDF if the 2nd CLI or HS2 was initialized before this UDF was created.
>
> Navis Ryu wrote:
> Permanent functions (persistent function seemed better name, imho) are registered to system registry, which is shared to all clients. So if one user creates new permanent function, it's shared to all clients. The time a user accesses the function, the class is loaded with required resources and registered to session registry as a temporary function.
>
> Jason Dere wrote:
> So this would work if all clients are using hiveserver2, because all clients in this scenario would share the same system registry.
> But if one or more clients are using the Hive CLI, any persistent UDFs created/dropped by this CLI client would not be reflected in the other clients (or HS2), since it's a different process/system registry.
I've missed this message. Yes, it will act like you've commented. But is it a common case to expect things done in separate VM refleceted to others? Should dropping presistent function by a jdbc client be reflected to other CLI clients?
- Navis
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review57952
-----------------------------------------------------------
On Oct. 30, 2014, 11:41 p.m., Jason Dere wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26854/
> -----------------------------------------------------------
>
> (Updated Oct. 30, 2014, 11:41 p.m.)
>
>
> Review request for hive, Navis Ryu and Thejas Nair.
>
>
> Bugs: HIVE-2573
> https://issues.apache.org/jira/browse/HIVE-2573
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Small updates to Navis' changes:
> - session registry doesn't lookup metastore for UDFs
> - my feedback from Navis' original patch
> - metastore udfs should not be considered native. This allows them to be added/removed from registry
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/common/JavaUtils.java 9aa917c
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7d8e5bc
> contrib/src/test/results/clientnegative/invalid_row_sequence.q.out 8f3c0b3
> itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithMiniKdc.java 6647ce5
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 88b0791
> ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 9ac540e
> ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java 074255b
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java e43a328
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
> ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 5bdeb92
> ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 4e3df75
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java b900627
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java 13277a9
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 211ab6c
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java e2768ff
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/translator/SqlFunctionConverter.java 793f117
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 1796b7b
> ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 22e5b47
> ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java 2b239ab
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionConf.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java af633cb
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
> ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
> ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out c7405ed
> ql/src/test/results/clientnegative/create_function_nonudf_class.q.out d0dd50a
> ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
> ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out e184787
> service/src/test/org/apache/hadoop/hive/service/TestHiveServerSessions.java fd38907
>
> Diff: https://reviews.apache.org/r/26854/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jason Dere
>
>
Re: Review Request 26854: HIVE-2573 Create per-session function
registry
Posted by Jason Dere <jd...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review57952
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java
<https://reviews.apache.org/r/26854/#comment98935>
Can we replace isNative/isPersistent with an enum that has BUILTIN, PERMANENT, TEMPORARY (or equivalent terms)?
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java
<https://reviews.apache.org/r/26854/#comment99013>
Why was this check removed? If you are permanent UDFs with Hive CLI, you would have to make sure the UDF resources are available from the cluster as opposed to just on the local filesystem of the client that created the UDF.
ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java
<https://reviews.apache.org/r/26854/#comment98987>
I thought builtin functions aren't allowed to be removed?
Does this mean that we could create a function using the same class as a built-in function (create a synonym), and deleting this new function will cause this class to be removed from the builtin set?
ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java
<https://reviews.apache.org/r/26854/#comment99016>
There is no longer a way to query the metastore for UDFs apart from the static initialization. So if one CLI user creates a permanent UDF, another user on CLI, or HS2, will not be able to use that new UDF if the 2nd CLI or HS2 was initialized before this UDF was created.
ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java
<https://reviews.apache.org/r/26854/#comment99009>
I think I see what you're trying do here, trying to add a mechanism so that if a function is deleted in one session, the other sessions will also see it as discarded if they try to look it up. But I don't actually see discarded being set to true.
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
<https://reviews.apache.org/r/26854/#comment98860>
I don't think it's necessary to pre-emptively query the metastore for permanent UDFs during initialization. If we have user on Hive CLI, we will automatically lookup metastore/download UDF resources, when they may not even be using any of these UDFs during their session. How about we keep the existing behavior that we only look them up when they are used during a query?
Also, if we are doing this during static initialization, is Hive be in a state that it can query the metastore? Not sure if there is any other initialization that may need to take place beforehand.
- Jason Dere
On Oct. 23, 2014, 12:20 a.m., Jason Dere wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26854/
> -----------------------------------------------------------
>
> (Updated Oct. 23, 2014, 12:20 a.m.)
>
>
> Review request for hive, Navis Ryu and Thejas Nair.
>
>
> Bugs: HIVE-2573
> https://issues.apache.org/jira/browse/HIVE-2573
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Small updates to Navis' changes:
> - session registry doesn't lookup metastore for UDFs
> - my feedback from Navis' original patch
> - metastore udfs should not be considered native. This allows them to be added/removed from registry
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 9ac540e
> ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java 074255b
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 08e1136
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
> ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java b900627
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/translator/SqlFunctionConverter.java 31f906a
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java e43d39f
> ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 22e5b47
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java af633cb
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
> ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
> ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out c7405ed
> ql/src/test/results/clientnegative/create_function_nonudf_class.q.out d0dd50a
> ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
> service/src/test/org/apache/hadoop/hive/service/TestHiveServerSessions.java fd38907
>
> Diff: https://reviews.apache.org/r/26854/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jason Dere
>
>
Re: Review Request 26854: HIVE-2573 Create per-session function
registry
Posted by Jason Dere <jd...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/
-----------------------------------------------------------
(Updated Oct. 23, 2014, 12:20 a.m.)
Review request for hive, Navis Ryu and Thejas Nair.
Changes
-------
Updating with HIVE-2573.9.patch.txt from Navis
Bugs: HIVE-2573
https://issues.apache.org/jira/browse/HIVE-2573
Repository: hive-git
Description
-------
Small updates to Navis' changes:
- session registry doesn't lookup metastore for UDFs
- my feedback from Navis' original patch
- metastore udfs should not be considered native. This allows them to be added/removed from registry
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 9ac540e
ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java 074255b
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 08e1136
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java b900627
ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/translator/SqlFunctionConverter.java 31f906a
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java e43d39f
ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 22e5b47
ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java af633cb
ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out c7405ed
ql/src/test/results/clientnegative/create_function_nonudf_class.q.out d0dd50a
ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
service/src/test/org/apache/hadoop/hive/service/TestHiveServerSessions.java fd38907
Diff: https://reviews.apache.org/r/26854/diff/
Testing
-------
Thanks,
Jason Dere
Re: Review Request 26854: HIVE-2573 Create per-session function
registry
Posted by Jason Dere <jd...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/
-----------------------------------------------------------
(Updated Oct. 20, 2014, 6:28 p.m.)
Review request for hive, Navis Ryu and Thejas Nair.
Changes
-------
Updating with Navis' changes - HIVE-2573.8.patch.txt - his description:
But one thing I want to change is that all functions in native registry should be native function and vise versa to clarify function types. I think we can define permanent function as a native function but acts as a temporary function when accessed, by registering itself to session registry. So it's,
system registry (native) : builtin-function + permanent function (contains definition only)
session register (non-native) : temporary function + permanent function
Bugs: HIVE-2573
https://issues.apache.org/jira/browse/HIVE-2573
Repository: hive-git
Description
-------
Small updates to Navis' changes:
- session registry doesn't lookup metastore for UDFs
- my feedback from Navis' original patch
- metastore udfs should not be considered native. This allows them to be added/removed from registry
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 9ac540e
ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java 074255b
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java b94f790
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/translator/SqlFunctionConverter.java 31f906a
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java e43d39f
ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 22e5b47
ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java af633cb
ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/MatchPath.java aa48a6c
ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/Noop.java fcf6afd
ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/NoopStreaming.java 41e0102
ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/NoopWithMap.java 0b090a9
ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/NoopWithMapStreaming.java 5d322d3
ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/WindowingTableFunction.java 903a9b0
ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out c7405ed
ql/src/test/results/clientnegative/create_function_nonudf_class.q.out d0dd50a
ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
service/src/test/org/apache/hadoop/hive/service/TestHiveServerSessions.java fd38907
Diff: https://reviews.apache.org/r/26854/diff/
Testing
-------
Thanks,
Jason Dere
Re: Review Request 26854: HIVE-2573 Create per-session function
registry
Posted by Thejas Nair <th...@hortonworks.com>.
> On Oct. 18, 2014, 5:34 p.m., Thejas Nair wrote:
> > ql/src/test/results/clientpositive/show_functions.q.out, line 21
> > <https://reviews.apache.org/r/26854/diff/1/?file=723916#file723916line21>
> >
> > we need to document that windowing functions show up with "@_" prefix in 'show_functions' output
>
> Lefty Leverenz wrote:
> Do you mean document in the wiki, or in javadocs?
>
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowFunctions
>
> Lefty Leverenz wrote:
> Also https://cwiki.apache.org/confluence/display/Hive/LanguageManual+WindowingAndAnalytics.
I mean the wiki docs.
You have very sharp eyes! I didn't think you would catch this review comment :)
- Thejas
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review57232
-----------------------------------------------------------
On Oct. 17, 2014, 2:21 a.m., Jason Dere wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26854/
> -----------------------------------------------------------
>
> (Updated Oct. 17, 2014, 2:21 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-2573
> https://issues.apache.org/jira/browse/HIVE-2573
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Small updates to Navis' changes:
> - session registry doesn't lookup metastore for UDFs
> - my feedback from Navis' original patch
> - metastore udfs should not be considered native. This allows them to be added/removed from registry
>
>
> Diffs
> -----
>
> contrib/src/test/results/clientnegative/invalid_row_sequence.q.out 8f3c0b3
> contrib/src/test/results/clientpositive/udaf_example_max.q.out 82aeca7
> contrib/src/test/results/clientpositive/udaf_example_min.q.out b62ff39
> itests/hive-unit/src/test/java/org/apache/hive/service/TestHiveServerSessions.java PRE-CREATION
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java a035ff1
> ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 31978fe
> ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java 074255b
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java b94f790
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
> ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
> ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 22e5b47
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java af633cb
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
> ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
> ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
> ql/src/test/results/clientpositive/show_functions.q.out 944c567
> ql/src/test/results/clientpositive/udaf_collect_set.q.out 42f2d9d
> ql/src/test/results/clientpositive/udaf_corr.q.out 1b468c4
> ql/src/test/results/clientpositive/udaf_covar_pop.q.out fa1d47e
> ql/src/test/results/clientpositive/udaf_covar_samp.q.out ff5b47e
> ql/src/test/results/clientpositive/udf_avg.q.out 7ac139f
> ql/src/test/results/clientpositive/udf_count.q.out 78228ec
> ql/src/test/results/clientpositive/udf_max.q.out 8535752
> ql/src/test/results/clientpositive/udf_min.q.out ca8158e
> ql/src/test/results/clientpositive/udf_percentile.q.out b963236
> ql/src/test/results/clientpositive/udf_std.q.out c004157
> ql/src/test/results/clientpositive/udf_stddev.q.out 8948dfc
> ql/src/test/results/clientpositive/udf_stddev_samp.q.out 6619ea1
> ql/src/test/results/clientpositive/udf_sum.q.out 2f4e3da
> ql/src/test/results/clientpositive/udf_var_pop.q.out cd9c61e
> ql/src/test/results/clientpositive/udf_var_samp.q.out 190435d
> ql/src/test/results/clientpositive/udf_variance.q.out c00dd66
> service/src/test/org/apache/hadoop/hive/service/TestHiveServerSessions.java fd38907
>
> Diff: https://reviews.apache.org/r/26854/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jason Dere
>
>
Re: Review Request 26854: HIVE-2573 Create per-session function
registry
Posted by Lefty Leverenz <le...@gmail.com>.
> On Oct. 18, 2014, 5:34 p.m., Thejas Nair wrote:
> > ql/src/test/results/clientpositive/show_functions.q.out, line 21
> > <https://reviews.apache.org/r/26854/diff/1/?file=723916#file723916line21>
> >
> > we need to document that windowing functions show up with "@_" prefix in 'show_functions' output
Do you mean document in the wiki, or in javadocs?
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowFunctions
- Lefty
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review57232
-----------------------------------------------------------
On Oct. 17, 2014, 2:21 a.m., Jason Dere wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26854/
> -----------------------------------------------------------
>
> (Updated Oct. 17, 2014, 2:21 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-2573
> https://issues.apache.org/jira/browse/HIVE-2573
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Small updates to Navis' changes:
> - session registry doesn't lookup metastore for UDFs
> - my feedback from Navis' original patch
> - metastore udfs should not be considered native. This allows them to be added/removed from registry
>
>
> Diffs
> -----
>
> contrib/src/test/results/clientnegative/invalid_row_sequence.q.out 8f3c0b3
> contrib/src/test/results/clientpositive/udaf_example_max.q.out 82aeca7
> contrib/src/test/results/clientpositive/udaf_example_min.q.out b62ff39
> itests/hive-unit/src/test/java/org/apache/hive/service/TestHiveServerSessions.java PRE-CREATION
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java a035ff1
> ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 31978fe
> ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java 074255b
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java b94f790
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
> ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
> ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 22e5b47
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java af633cb
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
> ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
> ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
> ql/src/test/results/clientpositive/show_functions.q.out 944c567
> ql/src/test/results/clientpositive/udaf_collect_set.q.out 42f2d9d
> ql/src/test/results/clientpositive/udaf_corr.q.out 1b468c4
> ql/src/test/results/clientpositive/udaf_covar_pop.q.out fa1d47e
> ql/src/test/results/clientpositive/udaf_covar_samp.q.out ff5b47e
> ql/src/test/results/clientpositive/udf_avg.q.out 7ac139f
> ql/src/test/results/clientpositive/udf_count.q.out 78228ec
> ql/src/test/results/clientpositive/udf_max.q.out 8535752
> ql/src/test/results/clientpositive/udf_min.q.out ca8158e
> ql/src/test/results/clientpositive/udf_percentile.q.out b963236
> ql/src/test/results/clientpositive/udf_std.q.out c004157
> ql/src/test/results/clientpositive/udf_stddev.q.out 8948dfc
> ql/src/test/results/clientpositive/udf_stddev_samp.q.out 6619ea1
> ql/src/test/results/clientpositive/udf_sum.q.out 2f4e3da
> ql/src/test/results/clientpositive/udf_var_pop.q.out cd9c61e
> ql/src/test/results/clientpositive/udf_var_samp.q.out 190435d
> ql/src/test/results/clientpositive/udf_variance.q.out c00dd66
> service/src/test/org/apache/hadoop/hive/service/TestHiveServerSessions.java fd38907
>
> Diff: https://reviews.apache.org/r/26854/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jason Dere
>
>
Re: Review Request 26854: HIVE-2573 Create per-session function
registry
Posted by Thejas Nair <th...@hortonworks.com>.
> On Oct. 18, 2014, 5:34 p.m., Thejas Nair wrote:
> > ql/src/test/results/clientpositive/show_functions.q.out, line 21
> > <https://reviews.apache.org/r/26854/diff/1/?file=723916#file723916line21>
> >
> > we need to document that windowing functions show up with "@_" prefix in 'show_functions' output
>
> Lefty Leverenz wrote:
> Do you mean document in the wiki, or in javadocs?
>
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowFunctions
>
> Lefty Leverenz wrote:
> Also https://cwiki.apache.org/confluence/display/Hive/LanguageManual+WindowingAndAnalytics.
>
> Thejas Nair wrote:
> I mean the wiki docs.
> You have very sharp eyes! I didn't think you would catch this review comment :)
>
> Lefty Leverenz wrote:
> <big grin> I'm so busy scanning email lists for doc issues, I don't have time to fix the doc issues.
>
> Jason Dere wrote:
> This was part of Navis' original changes .. hopefully he can explain why this is necessary?
I see why it is needed, you can have a regular udf and windowing udf with same name. So we need a way to distinguish between them in 'show functions' output
- Thejas
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review57232
-----------------------------------------------------------
On Oct. 20, 2014, 6:28 p.m., Jason Dere wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26854/
> -----------------------------------------------------------
>
> (Updated Oct. 20, 2014, 6:28 p.m.)
>
>
> Review request for hive, Navis Ryu and Thejas Nair.
>
>
> Bugs: HIVE-2573
> https://issues.apache.org/jira/browse/HIVE-2573
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Small updates to Navis' changes:
> - session registry doesn't lookup metastore for UDFs
> - my feedback from Navis' original patch
> - metastore udfs should not be considered native. This allows them to be added/removed from registry
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 9ac540e
> ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java 074255b
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java b94f790
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
> ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/translator/SqlFunctionConverter.java 31f906a
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java e43d39f
> ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 22e5b47
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java af633cb
> ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/MatchPath.java aa48a6c
> ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/Noop.java fcf6afd
> ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/NoopStreaming.java 41e0102
> ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/NoopWithMap.java 0b090a9
> ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/NoopWithMapStreaming.java 5d322d3
> ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/WindowingTableFunction.java 903a9b0
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
> ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
> ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out c7405ed
> ql/src/test/results/clientnegative/create_function_nonudf_class.q.out d0dd50a
> ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
> service/src/test/org/apache/hadoop/hive/service/TestHiveServerSessions.java fd38907
>
> Diff: https://reviews.apache.org/r/26854/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jason Dere
>
>
Re: Review Request 26854: HIVE-2573 Create per-session function
registry
Posted by Lefty Leverenz <le...@gmail.com>.
> On Oct. 18, 2014, 5:34 p.m., Thejas Nair wrote:
> > ql/src/test/results/clientpositive/show_functions.q.out, line 21
> > <https://reviews.apache.org/r/26854/diff/1/?file=723916#file723916line21>
> >
> > we need to document that windowing functions show up with "@_" prefix in 'show_functions' output
>
> Lefty Leverenz wrote:
> Do you mean document in the wiki, or in javadocs?
>
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowFunctions
Also https://cwiki.apache.org/confluence/display/Hive/LanguageManual+WindowingAndAnalytics.
- Lefty
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review57232
-----------------------------------------------------------
On Oct. 17, 2014, 2:21 a.m., Jason Dere wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26854/
> -----------------------------------------------------------
>
> (Updated Oct. 17, 2014, 2:21 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-2573
> https://issues.apache.org/jira/browse/HIVE-2573
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Small updates to Navis' changes:
> - session registry doesn't lookup metastore for UDFs
> - my feedback from Navis' original patch
> - metastore udfs should not be considered native. This allows them to be added/removed from registry
>
>
> Diffs
> -----
>
> contrib/src/test/results/clientnegative/invalid_row_sequence.q.out 8f3c0b3
> contrib/src/test/results/clientpositive/udaf_example_max.q.out 82aeca7
> contrib/src/test/results/clientpositive/udaf_example_min.q.out b62ff39
> itests/hive-unit/src/test/java/org/apache/hive/service/TestHiveServerSessions.java PRE-CREATION
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java a035ff1
> ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 31978fe
> ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java 074255b
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java b94f790
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
> ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
> ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 22e5b47
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java af633cb
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
> ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
> ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
> ql/src/test/results/clientpositive/show_functions.q.out 944c567
> ql/src/test/results/clientpositive/udaf_collect_set.q.out 42f2d9d
> ql/src/test/results/clientpositive/udaf_corr.q.out 1b468c4
> ql/src/test/results/clientpositive/udaf_covar_pop.q.out fa1d47e
> ql/src/test/results/clientpositive/udaf_covar_samp.q.out ff5b47e
> ql/src/test/results/clientpositive/udf_avg.q.out 7ac139f
> ql/src/test/results/clientpositive/udf_count.q.out 78228ec
> ql/src/test/results/clientpositive/udf_max.q.out 8535752
> ql/src/test/results/clientpositive/udf_min.q.out ca8158e
> ql/src/test/results/clientpositive/udf_percentile.q.out b963236
> ql/src/test/results/clientpositive/udf_std.q.out c004157
> ql/src/test/results/clientpositive/udf_stddev.q.out 8948dfc
> ql/src/test/results/clientpositive/udf_stddev_samp.q.out 6619ea1
> ql/src/test/results/clientpositive/udf_sum.q.out 2f4e3da
> ql/src/test/results/clientpositive/udf_var_pop.q.out cd9c61e
> ql/src/test/results/clientpositive/udf_var_samp.q.out 190435d
> ql/src/test/results/clientpositive/udf_variance.q.out c00dd66
> service/src/test/org/apache/hadoop/hive/service/TestHiveServerSessions.java fd38907
>
> Diff: https://reviews.apache.org/r/26854/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jason Dere
>
>
Re: Review Request 26854: HIVE-2573 Create per-session function
registry
Posted by Jason Dere <jd...@hortonworks.com>.
> On Oct. 18, 2014, 5:34 p.m., Thejas Nair wrote:
> > itests/hive-unit/src/test/java/org/apache/hive/service/TestHiveServerSessions.java, line 61
> > <https://reviews.apache.org/r/26854/diff/1/?file=723902#file723902line61>
> >
> > should we use hiveserver2 (/minish2) instead since that is the commomly used server ? HiveServer might go away in some time.
This is actually an existing test, I simply moved it from service/ to itests/ and added an extra test case.
Could look into changing to hiveserver2 if I have time .. do you have an example test I can refer to?
> On Oct. 18, 2014, 5:34 p.m., Thejas Nair wrote:
> > ql/src/test/results/clientpositive/show_functions.q.out, line 21
> > <https://reviews.apache.org/r/26854/diff/1/?file=723916#file723916line21>
> >
> > we need to document that windowing functions show up with "@_" prefix in 'show_functions' output
>
> Lefty Leverenz wrote:
> Do you mean document in the wiki, or in javadocs?
>
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowFunctions
>
> Lefty Leverenz wrote:
> Also https://cwiki.apache.org/confluence/display/Hive/LanguageManual+WindowingAndAnalytics.
>
> Thejas Nair wrote:
> I mean the wiki docs.
> You have very sharp eyes! I didn't think you would catch this review comment :)
>
> Lefty Leverenz wrote:
> <big grin> I'm so busy scanning email lists for doc issues, I don't have time to fix the doc issues.
This was part of Navis' original changes .. hopefully he can explain why this is necessary?
- Jason
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review57232
-----------------------------------------------------------
On Oct. 20, 2014, 6:28 p.m., Jason Dere wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26854/
> -----------------------------------------------------------
>
> (Updated Oct. 20, 2014, 6:28 p.m.)
>
>
> Review request for hive, Navis Ryu and Thejas Nair.
>
>
> Bugs: HIVE-2573
> https://issues.apache.org/jira/browse/HIVE-2573
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Small updates to Navis' changes:
> - session registry doesn't lookup metastore for UDFs
> - my feedback from Navis' original patch
> - metastore udfs should not be considered native. This allows them to be added/removed from registry
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 9ac540e
> ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java 074255b
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java b94f790
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
> ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/translator/SqlFunctionConverter.java 31f906a
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java e43d39f
> ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 22e5b47
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java af633cb
> ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/MatchPath.java aa48a6c
> ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/Noop.java fcf6afd
> ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/NoopStreaming.java 41e0102
> ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/NoopWithMap.java 0b090a9
> ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/NoopWithMapStreaming.java 5d322d3
> ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/WindowingTableFunction.java 903a9b0
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
> ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
> ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out c7405ed
> ql/src/test/results/clientnegative/create_function_nonudf_class.q.out d0dd50a
> ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
> service/src/test/org/apache/hadoop/hive/service/TestHiveServerSessions.java fd38907
>
> Diff: https://reviews.apache.org/r/26854/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jason Dere
>
>
Re: Review Request 26854: HIVE-2573 Create per-session function
registry
Posted by Lefty Leverenz <le...@gmail.com>.
> On Oct. 18, 2014, 5:34 p.m., Thejas Nair wrote:
> > ql/src/test/results/clientpositive/show_functions.q.out, line 21
> > <https://reviews.apache.org/r/26854/diff/1/?file=723916#file723916line21>
> >
> > we need to document that windowing functions show up with "@_" prefix in 'show_functions' output
>
> Lefty Leverenz wrote:
> Do you mean document in the wiki, or in javadocs?
>
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowFunctions
>
> Lefty Leverenz wrote:
> Also https://cwiki.apache.org/confluence/display/Hive/LanguageManual+WindowingAndAnalytics.
>
> Thejas Nair wrote:
> I mean the wiki docs.
> You have very sharp eyes! I didn't think you would catch this review comment :)
<big grin> I'm so busy scanning email lists for doc issues, I don't have time to fix the doc issues.
- Lefty
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review57232
-----------------------------------------------------------
On Oct. 17, 2014, 2:21 a.m., Jason Dere wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26854/
> -----------------------------------------------------------
>
> (Updated Oct. 17, 2014, 2:21 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-2573
> https://issues.apache.org/jira/browse/HIVE-2573
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Small updates to Navis' changes:
> - session registry doesn't lookup metastore for UDFs
> - my feedback from Navis' original patch
> - metastore udfs should not be considered native. This allows them to be added/removed from registry
>
>
> Diffs
> -----
>
> contrib/src/test/results/clientnegative/invalid_row_sequence.q.out 8f3c0b3
> contrib/src/test/results/clientpositive/udaf_example_max.q.out 82aeca7
> contrib/src/test/results/clientpositive/udaf_example_min.q.out b62ff39
> itests/hive-unit/src/test/java/org/apache/hive/service/TestHiveServerSessions.java PRE-CREATION
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java a035ff1
> ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 31978fe
> ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java 074255b
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java b94f790
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
> ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
> ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 22e5b47
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java af633cb
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
> ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
> ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
> ql/src/test/results/clientpositive/show_functions.q.out 944c567
> ql/src/test/results/clientpositive/udaf_collect_set.q.out 42f2d9d
> ql/src/test/results/clientpositive/udaf_corr.q.out 1b468c4
> ql/src/test/results/clientpositive/udaf_covar_pop.q.out fa1d47e
> ql/src/test/results/clientpositive/udaf_covar_samp.q.out ff5b47e
> ql/src/test/results/clientpositive/udf_avg.q.out 7ac139f
> ql/src/test/results/clientpositive/udf_count.q.out 78228ec
> ql/src/test/results/clientpositive/udf_max.q.out 8535752
> ql/src/test/results/clientpositive/udf_min.q.out ca8158e
> ql/src/test/results/clientpositive/udf_percentile.q.out b963236
> ql/src/test/results/clientpositive/udf_std.q.out c004157
> ql/src/test/results/clientpositive/udf_stddev.q.out 8948dfc
> ql/src/test/results/clientpositive/udf_stddev_samp.q.out 6619ea1
> ql/src/test/results/clientpositive/udf_sum.q.out 2f4e3da
> ql/src/test/results/clientpositive/udf_var_pop.q.out cd9c61e
> ql/src/test/results/clientpositive/udf_var_samp.q.out 190435d
> ql/src/test/results/clientpositive/udf_variance.q.out c00dd66
> service/src/test/org/apache/hadoop/hive/service/TestHiveServerSessions.java fd38907
>
> Diff: https://reviews.apache.org/r/26854/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jason Dere
>
>
Re: Review Request 26854: HIVE-2573 Create per-session function
registry
Posted by Thejas Nair <th...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/#review57232
-----------------------------------------------------------
itests/hive-unit/src/test/java/org/apache/hive/service/TestHiveServerSessions.java
<https://reviews.apache.org/r/26854/#comment97820>
should we use hiveserver2 (/minish2) instead since that is the commomly used server ? HiveServer might go away in some time.
ql/src/test/results/clientpositive/show_functions.q.out
<https://reviews.apache.org/r/26854/#comment97819>
we need to document that windowing functions show up with "@_" prefix in 'show_functions' output
- Thejas Nair
On Oct. 17, 2014, 2:21 a.m., Jason Dere wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26854/
> -----------------------------------------------------------
>
> (Updated Oct. 17, 2014, 2:21 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-2573
> https://issues.apache.org/jira/browse/HIVE-2573
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Small updates to Navis' changes:
> - session registry doesn't lookup metastore for UDFs
> - my feedback from Navis' original patch
> - metastore udfs should not be considered native. This allows them to be added/removed from registry
>
>
> Diffs
> -----
>
> contrib/src/test/results/clientnegative/invalid_row_sequence.q.out 8f3c0b3
> contrib/src/test/results/clientpositive/udaf_example_max.q.out 82aeca7
> contrib/src/test/results/clientpositive/udaf_example_min.q.out b62ff39
> itests/hive-unit/src/test/java/org/apache/hive/service/TestHiveServerSessions.java PRE-CREATION
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java a035ff1
> ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 31978fe
> ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java 074255b
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java b94f790
> ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
> ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
> ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 22e5b47
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java af633cb
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
> ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
> ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
> ql/src/test/results/clientpositive/show_functions.q.out 944c567
> ql/src/test/results/clientpositive/udaf_collect_set.q.out 42f2d9d
> ql/src/test/results/clientpositive/udaf_corr.q.out 1b468c4
> ql/src/test/results/clientpositive/udaf_covar_pop.q.out fa1d47e
> ql/src/test/results/clientpositive/udaf_covar_samp.q.out ff5b47e
> ql/src/test/results/clientpositive/udf_avg.q.out 7ac139f
> ql/src/test/results/clientpositive/udf_count.q.out 78228ec
> ql/src/test/results/clientpositive/udf_max.q.out 8535752
> ql/src/test/results/clientpositive/udf_min.q.out ca8158e
> ql/src/test/results/clientpositive/udf_percentile.q.out b963236
> ql/src/test/results/clientpositive/udf_std.q.out c004157
> ql/src/test/results/clientpositive/udf_stddev.q.out 8948dfc
> ql/src/test/results/clientpositive/udf_stddev_samp.q.out 6619ea1
> ql/src/test/results/clientpositive/udf_sum.q.out 2f4e3da
> ql/src/test/results/clientpositive/udf_var_pop.q.out cd9c61e
> ql/src/test/results/clientpositive/udf_var_samp.q.out 190435d
> ql/src/test/results/clientpositive/udf_variance.q.out c00dd66
> service/src/test/org/apache/hadoop/hive/service/TestHiveServerSessions.java fd38907
>
> Diff: https://reviews.apache.org/r/26854/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jason Dere
>
>
Re: Review Request 26854: HIVE-2573 Create per-session function
registry
Posted by Jason Dere <jd...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26854/
-----------------------------------------------------------
(Updated Oct. 17, 2014, 2:21 a.m.)
Review request for hive.
Bugs: HIVE-2573
https://issues.apache.org/jira/browse/HIVE-2573
Repository: hive-git
Description
-------
Small updates to Navis' changes:
- session registry doesn't lookup metastore for UDFs
- my feedback from Navis' original patch
- metastore udfs should not be considered native. This allows them to be added/removed from registry
Diffs
-----
contrib/src/test/results/clientnegative/invalid_row_sequence.q.out 8f3c0b3
contrib/src/test/results/clientpositive/udaf_example_max.q.out 82aeca7
contrib/src/test/results/clientpositive/udaf_example_min.q.out b62ff39
itests/hive-unit/src/test/java/org/apache/hive/service/TestHiveServerSessions.java PRE-CREATION
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java a035ff1
ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 31978fe
ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java 93c15c0
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java 074255b
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java b94f790
ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 569c125
ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java efecb05
ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java 22e5b47
ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java af633cb
ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 46f8052
ql/src/test/queries/clientnegative/drop_native_udf.q ae047bb
ql/src/test/results/clientnegative/drop_native_udf.q.out 9f0eaa5
ql/src/test/results/clientpositive/show_functions.q.out 944c567
ql/src/test/results/clientpositive/udaf_collect_set.q.out 42f2d9d
ql/src/test/results/clientpositive/udaf_corr.q.out 1b468c4
ql/src/test/results/clientpositive/udaf_covar_pop.q.out fa1d47e
ql/src/test/results/clientpositive/udaf_covar_samp.q.out ff5b47e
ql/src/test/results/clientpositive/udf_avg.q.out 7ac139f
ql/src/test/results/clientpositive/udf_count.q.out 78228ec
ql/src/test/results/clientpositive/udf_max.q.out 8535752
ql/src/test/results/clientpositive/udf_min.q.out ca8158e
ql/src/test/results/clientpositive/udf_percentile.q.out b963236
ql/src/test/results/clientpositive/udf_std.q.out c004157
ql/src/test/results/clientpositive/udf_stddev.q.out 8948dfc
ql/src/test/results/clientpositive/udf_stddev_samp.q.out 6619ea1
ql/src/test/results/clientpositive/udf_sum.q.out 2f4e3da
ql/src/test/results/clientpositive/udf_var_pop.q.out cd9c61e
ql/src/test/results/clientpositive/udf_var_samp.q.out 190435d
ql/src/test/results/clientpositive/udf_variance.q.out c00dd66
service/src/test/org/apache/hadoop/hive/service/TestHiveServerSessions.java fd38907
Diff: https://reviews.apache.org/r/26854/diff/
Testing
-------
Thanks,
Jason Dere