[jira] [Commented] (HIVE-2627) NPE on MAP-JOIN with a UDF in an external JAR

["dima machlin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-2627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14009503#comment-14009503 ] 

dima machlin commented on HIVE-2627:
------------------------------------

I can confirm that this still happens in hive 0.12.
Getting : 

java.lang.ClassNotFoundException: com.some.class.used.by.UDF
Continuing ...
java.lang.NullPointerException: target should not be null
java.lang.NullPointerException: target should not be null
Continuing ...

and eventually 

ERROR mr.MapredLocalTask: Hive Runtime Error: Map local work failed
java.lang.NullPointerException
        at org.apache.hadoop.hive.ql.exec.FunctionRegistry.isStateful(FunctionRegistry.java:1415)
        at org.apache.hadoop.hive.ql.exec.FunctionRegistry.isDeterministic(FunctionRegistry.java:1385)
        at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.isDeterministic(ExprNodeGenericFuncEvaluator.java:132)
        at org.apache.hadoop.hive.ql.exec.FunctionRegistry.isDeterministic(FunctionRegistry.java:1385)
        at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.isDeterministic(ExprNodeGenericFuncEvaluator.java:132)
        at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.iterate(ExprNodeEvaluatorFactory.java:83)    at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.iterate(ExprNodeEvaluatorFactory.java:83)
        at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.toCachedEval(ExprNodeEvaluatorFactory.java:73)
        at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:57)
        at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:57)
        at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:377)        at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:377)

        at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:453)        at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:453)
        at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:409)
        at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:188)
        at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:188)
        at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:377)
        at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initializeOperators(MapredLocalTask.java:419)
        at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeFromChildJVM(MapredLocalTask.java:305)
        at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:722)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:197)



> NPE on MAP-JOIN with a UDF in an external JAR
> ---------------------------------------------
>
>                 Key: HIVE-2627
>                 URL: https://issues.apache.org/jira/browse/HIVE-2627
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Jonathan Chang
>
> When a query is converted into a map join, and it depends on some UDF (ADD JAR...; CREATE TEMPORARY FUNCTION...), then an NPE may happen.  Here is an example.
> SELECT
>     some_udf(dummy1) as dummies
> FROM (
>     SELECT        
>         a.dummy as dummy1,
>         b.dummy as dummy2
>     FROM        
>         test a    
>     LEFT OUTER JOIN
>         test b
>     ON
>         a.dummy = b.dummy
> ) c;
> My guess is that the JAR classes are not getting propagated to the hashmapjoin operator.



--
This message was sent by Atlassian JIRA
(v6.2#6252)