You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2019/03/01 21:51:00 UTC

[jira] [Resolved] (SPARK-26624) Different classloader use on subsequent call to same query, causing different behavior

     [ https://issues.apache.org/jira/browse/SPARK-26624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen resolved SPARK-26624.
-------------------------------
    Resolution: Duplicate

> Different classloader use on subsequent call to same query, causing different behavior
> --------------------------------------------------------------------------------------
>
>                 Key: SPARK-26624
>                 URL: https://issues.apache.org/jira/browse/SPARK-26624
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.2.0
>            Reporter: Bimalendu Choudhary
>            Priority: Major
>
> When calling a Hive UDF function from spark shell, we get the output when we call the query first time, but when we call the query again it gives following error
> #spark2-shell
> scala> spark.sql("select test(name) from customers limit 2").show (50, false)
> org.apache.spark.sql.AnalysisException: No handler for Hive UDF 'com.vnb.fgp.generic.udf.encrypt.EncryptGenericUDF': 
> We have not provided the UDF jar files on the command line, but still we get the output. The function test is created in Hive service as a permanent function using the jar file.
> Debugging it further we see that on first invocation of the select command the  following classLoader is being used and it has a path pointing to the hdfs directory as set in Hive service:
> loader:  org.apache.spark.sql.internal.NonClosableMutableURLClassLoader@42cef0af
> hdfs:/tmp/bimal/hive-extensions-1.0-SNAPSHOT-jar-with-dependencies.jar
> file:/usr/java/jdk1.8.0_162/jre/lib/resources.jar
> file:/usr/java/jdk1.8.0_162/jre/lib/rt.jar
> On subsequent calls, a different class loader is being used:
> loader scala.tools.nsc.interpreter.IMain$TranslatingClassLoader@7bc3ec95
> file:/usr/java/jdk1.8.0_162/jre/lib/resources.jar
> file:/usr/java/jdk1.8.0_162/jre/lib/rt.jar
> file:/usr/java/jdk1.8.0_162/jre/lib/jsse.jar
> file:/usr/java/jdk1.8.0_162/jre/lib/jce.jar
> This does not have the hdfs path for the jar file and hence the exception is generated.
> Most probably the classloader is picking things from Hive metastore.
> If we pass the UDF jar files on command line using --jars option, everything works fine. 
> But this indicates that the classLoader and classpaths are different when called first and second time causing inconsistent behavior and cause problem.
>   



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org