You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2020/07/13 08:55:07 UTC
[GitHub] [hive] dengzhhu653 opened a new pull request #1242: HIVE-20441: NPE in GenericUDF when hive.allow.udf.load.on.demand is s…
dengzhhu653 opened a new pull request #1242:
URL: https://github.com/apache/hive/pull/1242
…et to true
## NOTICE
Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HIVE-XXXXX: Fix a typo in YYY)
For more details, please see https://cwiki.apache.org/confluence/display/Hive/HowToContribute
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on a change in pull request #1242: HIVE-20441: NPE in GenericUDF when hive.allow.udf.load.on.demand is s…
Posted by GitBox <gi...@apache.org>.
dengzhhu653 commented on a change in pull request #1242:
URL: https://github.com/apache/hive/pull/1242#discussion_r457248607
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java
##########
@@ -293,9 +293,10 @@ public FunctionInfo registerPermanentFunction(String functionName,
if (registerToSession) {
String qualifiedName = FunctionUtils.qualifyFunctionName(
functionName, SessionState.get().getCurrentDatabase().toLowerCase());
- if (registerToSessionRegistry(qualifiedName, function) != null) {
+ FunctionInfo newFunction = registerToSessionRegistry(qualifiedName, function);
Review comment:
The call ```FunctionRegistry.getFunctionInfo(String functionName)``` will make HS2 will lookup the function from MetaStore when the function does not find in the session or system registry with hive.allow.udf.load.on.demand enabled. If the function is found, a FunctionInfo created by ```new FunctionInfo(functionName, className, resources)``` will be returned, but the genericUDF field of the FunctionInfo is null,
https://github.com/apache/hive/blob/fa086ecce543384993a61d5154b14fa3b80df3b5/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java#L67-L74 .
So when TypeCheckProcFactory.DefaultExprProcessor gets function expr from AstNode,
https://github.com/apache/hive/blob/fa086ecce543384993a61d5154b14fa3b80df3b5/ql/src/java/org/apache/hadoop/hive/ql/parse/type/TypeCheckProcFactory.java#L935-L948, The genericUDF got from ```GenericUDF genericUDF = fi.getGenericUDF();``` is null, if the genericUDF is used to create function expr desc afterwards, a npe will be thrown.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on pull request #1242: HIVE-20441: NPE in GenericUDF when hive.allow.udf.load.on.demand is s…
Posted by GitBox <gi...@apache.org>.
dengzhhu653 commented on pull request #1242:
URL: https://github.com/apache/hive/pull/1242#issuecomment-657894768
The failed tests seem unrelated...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on a change in pull request #1242: HIVE-20441: NPE in GenericUDF when hive.allow.udf.load.on.demand is s…
Posted by GitBox <gi...@apache.org>.
dengzhhu653 commented on a change in pull request #1242:
URL: https://github.com/apache/hive/pull/1242#discussion_r457248607
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java
##########
@@ -293,9 +293,10 @@ public FunctionInfo registerPermanentFunction(String functionName,
if (registerToSession) {
String qualifiedName = FunctionUtils.qualifyFunctionName(
functionName, SessionState.get().getCurrentDatabase().toLowerCase());
- if (registerToSessionRegistry(qualifiedName, function) != null) {
+ FunctionInfo newFunction = registerToSessionRegistry(qualifiedName, function);
Review comment:
The call ```FunctionRegistry.getFunctionInfo(String functionName)``` will make HS2 will lookup the function from MetaStore when the function does not find in the session or system registry with hive.allow.udf.load.on.demand enabled. If the function is found, a FunctionInfo created by ```new FunctionInfo(functionName, className, resources)``` will be returned, but the genericUDF field of the FunctionInfo is null,
https://github.com/apache/hive/blob/fa086ecce543384993a61d5154b14fa3b80df3b5/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java#L67-L74 .
So when TypeCheckProcFactory.DefaultExprProcessor gets function expr from AstNode,
https://github.com/apache/hive/blob/fa086ecce543384993a61d5154b14fa3b80df3b5/ql/src/java/org/apache/hadoop/hive/ql/parse/type/TypeCheckProcFactory.java#L935-L948, The genericUDF got from ```GenericUDF genericUDF = fi.getGenericUDF();``` is null, if the genericUDF is used to create function expr desc afterwards, a npe will be thrown.
https://github.com/apache/hive/blob/fa086ecce543384993a61d5154b14fa3b80df3b5/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java#L117-L123
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 closed pull request #1242: HIVE-20441: NPE in GenericUDF when hive.allow.udf.load.on.demand is s…
Posted by GitBox <gi...@apache.org>.
dengzhhu653 closed pull request #1242:
URL: https://github.com/apache/hive/pull/1242
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on pull request #1242: HIVE-20441: NPE in GenericUDF when hive.allow.udf.load.on.demand is s…
Posted by GitBox <gi...@apache.org>.
dengzhhu653 commented on pull request #1242:
URL: https://github.com/apache/hive/pull/1242#issuecomment-668366427
Hi @pvary, Is there anything I can do to move this forward?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary commented on a change in pull request #1242: HIVE-20441: NPE in GenericUDF when hive.allow.udf.load.on.demand is s…
Posted by GitBox <gi...@apache.org>.
pvary commented on a change in pull request #1242:
URL: https://github.com/apache/hive/pull/1242#discussion_r457202709
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java
##########
@@ -293,9 +293,10 @@ public FunctionInfo registerPermanentFunction(String functionName,
if (registerToSession) {
String qualifiedName = FunctionUtils.qualifyFunctionName(
functionName, SessionState.get().getCurrentDatabase().toLowerCase());
- if (registerToSessionRegistry(qualifiedName, function) != null) {
+ FunctionInfo newFunction = registerToSessionRegistry(qualifiedName, function);
Review comment:
I tried to understand the goal of the change, but could not find the root cause, and do not see what I miss.
Could you please explain when this code results in different return value before and after the patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 closed pull request #1242: HIVE-20441: NPE in GenericUDF when hive.allow.udf.load.on.demand is s…
Posted by GitBox <gi...@apache.org>.
dengzhhu653 closed pull request #1242:
URL: https://github.com/apache/hive/pull/1242
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on a change in pull request #1242: HIVE-20441: NPE in GenericUDF when hive.allow.udf.load.on.demand is s…
Posted by GitBox <gi...@apache.org>.
dengzhhu653 commented on a change in pull request #1242:
URL: https://github.com/apache/hive/pull/1242#discussion_r457255692
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java
##########
@@ -293,9 +293,10 @@ public FunctionInfo registerPermanentFunction(String functionName,
if (registerToSession) {
String qualifiedName = FunctionUtils.qualifyFunctionName(
functionName, SessionState.get().getCurrentDatabase().toLowerCase());
- if (registerToSessionRegistry(qualifiedName, function) != null) {
+ FunctionInfo newFunction = registerToSessionRegistry(qualifiedName, function);
Review comment:
```registerToSessionRegistry``` will finally initialize the genericUDF by calling ```FunctionInfo(FunctionType functionType, String displayName, GenericUDF genericUDF, FunctionResource... resources)``` https://github.com/apache/hive/blob/fa086ecce543384993a61d5154b14fa3b80df3b5/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java#L76-L83, The genericUDF field of ```FunctionInfo``` returned by this call would not be null.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary merged pull request #1242: HIVE-20441: NPE in GenericUDF when hive.allow.udf.load.on.demand is s…
Posted by GitBox <gi...@apache.org>.
pvary merged pull request #1242:
URL: https://github.com/apache/hive/pull/1242
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on a change in pull request #1242: HIVE-20441: NPE in GenericUDF when hive.allow.udf.load.on.demand is s…
Posted by GitBox <gi...@apache.org>.
dengzhhu653 commented on a change in pull request #1242:
URL: https://github.com/apache/hive/pull/1242#discussion_r457248607
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java
##########
@@ -293,9 +293,10 @@ public FunctionInfo registerPermanentFunction(String functionName,
if (registerToSession) {
String qualifiedName = FunctionUtils.qualifyFunctionName(
functionName, SessionState.get().getCurrentDatabase().toLowerCase());
- if (registerToSessionRegistry(qualifiedName, function) != null) {
+ FunctionInfo newFunction = registerToSessionRegistry(qualifiedName, function);
Review comment:
The call ```FunctionRegistry.getFunctionInfo(String functionName)``` will make HS2 will lookup the function from MetaStore when the function does not find in the session or system registry with hive.allow.udf.load.on.demand enabled. If the function is found, a FunctionInfo created by ```new FunctionInfo(functionName, className, resources)``` will be returned, but the genericUDF field of the FunctionInfo is null, https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java#L67-L74 . So when TypeCheckProcFactory.DefaultExprProcessor gets function expr from AstNode, https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/type/TypeCheckProcFactory.java#L935-L948, The genericUDF got from ```GenericUDF genericUDF = fi.getGenericUDF();``` is null, if the genericUDF is used to create function expr desc afterwards, a npe will be thrown.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on pull request #1242: HIVE-20441: NPE in GenericUDF when hive.allow.udf.load.on.demand is s…
Posted by GitBox <gi...@apache.org>.
dengzhhu653 commented on pull request #1242:
URL: https://github.com/apache/hive/pull/1242#issuecomment-660384821
@kgyrtkirk @pvary cloud you please take a look? thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org