You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Chakravarthi (JIRA)" <ji...@apache.org> on 2019/03/05 13:53:00 UTC
[jira] [Comment Edited] (SPARK-26602) Insert into table fails after
querying the UDF which is loaded with wrong hdfs path
[ https://issues.apache.org/jira/browse/SPARK-26602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16784134#comment-16784134 ]
Chakravarthi edited comment on SPARK-26602 at 3/5/19 1:52 PM:
--------------------------------------------------------------
Hi [~srowen] , this issue is not duplicate of SPARK-26560. Here the issue is,Insert into table fails after querying the UDF which is loaded with wrong hdfs path.
Below are the steps to reproduce this issue:
1) create a table.
sql("create table table1(I int)");
2) create udf using invalid hdfs path.
sql("CREATE FUNCTION before_fix AS 'org.apache.hadoop.hive.ql.udf.generic.GenericUDFLastDayTest' USING JAR 'hdfs:///tmp/notexist.jar'")
3) Do select on the UDF and you will get exception as "Failed to read external resource".
sql(" select before_fix('2018-03-09')").
4) perform insert table.
sql("insert into table1 values(1)").show
Here ,insert should work.but is fails.
was (Author: chakravarthi):
Hi [~srowen] , this issue is not duplicate of SPARK-26560. Here the issue is,Insert into table fails after querying the UDF which is loaded with wrong hdfs path.
Below are the steps to reproduce this issue:
1) create a table.
sql("create table check_udf(I int)");
2) create udf using invalid hdfs path.
sql("CREATE FUNCTION before_fix AS 'org.apache.hadoop.hive.ql.udf.generic.GenericUDFLastDayTest' USING JAR 'hdfs:///tmp/notexist.jar'")
3) Do select on the UDF and you will get exception as "Failed to read external resource".
sql(" select before_fix('2018-03-09')").
4) perform insert table.
sql("insert into check_udf values(1)").show
Here ,insert should work.but is fails.
> Insert into table fails after querying the UDF which is loaded with wrong hdfs path
> -----------------------------------------------------------------------------------
>
> Key: SPARK-26602
> URL: https://issues.apache.org/jira/browse/SPARK-26602
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.0
> Reporter: Haripriya
> Priority: Major
> Attachments: beforeFixUdf.txt
>
>
> In sql,
> 1.Query the existing udf(say myFunc1)
> 2. create and select the udf registered with incorrect path (say myFunc2)
> 3.Now again query the existing udf in the same session - Wil throw exception stating that couldn't read resource of myFunc2's path
> 4.Even the basic operations like insert and select will fail giving the same error
> Result:
> java.lang.RuntimeException: Failed to read external resource hdfs:///tmp/hari_notexists1/two_udfs.jar
> at org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1288)
> at org.apache.hadoop.hive.ql.session.SessionState.resolveAndDownload(SessionState.java:1242)
> at org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1163)
> at org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1149)
> at org.apache.hadoop.hive.ql.processors.AddResourceProcessor.run(AddResourceProcessor.java:67)
> at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:737)
> at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:706)
> at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:275)
> at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:213)
> at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:212)
> at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:258)
> at org.apache.spark.sql.hive.client.HiveClientImpl.runHive(HiveClientImpl.scala:706)
> at org.apache.spark.sql.hive.client.HiveClientImpl.runSqlHive(HiveClientImpl.scala:696)
> at org.apache.spark.sql.hive.client.HiveClientImpl.addJar(HiveClientImpl.scala:841)
> at org.apache.spark.sql.hive.HiveSessionResourceLoader.addJar(HiveSessionStateBuilder.scala:112)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org