You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2019/03/01 21:51:00 UTC
[jira] [Resolved] (SPARK-26624) Different classloader use on
subsequent call to same query, causing different behavior
[ https://issues.apache.org/jira/browse/SPARK-26624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-26624.
-------------------------------
Resolution: Duplicate
> Different classloader use on subsequent call to same query, causing different behavior
> --------------------------------------------------------------------------------------
>
> Key: SPARK-26624
> URL: https://issues.apache.org/jira/browse/SPARK-26624
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: Bimalendu Choudhary
> Priority: Major
>
> When calling a Hive UDF function from spark shell, we get the output when we call the query first time, but when we call the query again it gives following error
> #spark2-shell
> scala> spark.sql("select test(name) from customers limit 2").show (50, false)
> org.apache.spark.sql.AnalysisException: No handler for Hive UDF 'com.vnb.fgp.generic.udf.encrypt.EncryptGenericUDF':
> We have not provided the UDF jar files on the command line, but still we get the output. The function test is created in Hive service as a permanent function using the jar file.
> Debugging it further we see that on first invocation of the select command the following classLoader is being used and it has a path pointing to the hdfs directory as set in Hive service:
> loader: org.apache.spark.sql.internal.NonClosableMutableURLClassLoader@42cef0af
> hdfs:/tmp/bimal/hive-extensions-1.0-SNAPSHOT-jar-with-dependencies.jar
> file:/usr/java/jdk1.8.0_162/jre/lib/resources.jar
> file:/usr/java/jdk1.8.0_162/jre/lib/rt.jar
> On subsequent calls, a different class loader is being used:
> loader scala.tools.nsc.interpreter.IMain$TranslatingClassLoader@7bc3ec95
> file:/usr/java/jdk1.8.0_162/jre/lib/resources.jar
> file:/usr/java/jdk1.8.0_162/jre/lib/rt.jar
> file:/usr/java/jdk1.8.0_162/jre/lib/jsse.jar
> file:/usr/java/jdk1.8.0_162/jre/lib/jce.jar
> This does not have the hdfs path for the jar file and hence the exception is generated.
> Most probably the classloader is picking things from Hive metastore.
> If we pass the UDF jar files on command line using --jars option, everything works fine.
> But this indicates that the classLoader and classpaths are different when called first and second time causing inconsistent behavior and cause problem.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org