You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Takeshi Yamamuro (JIRA)" <ji...@apache.org> on 2017/03/24 11:39:41 UTC
[jira] [Commented] (SPARK-8586) SQL add jar command does not work
well with Scala REPL
[ https://issues.apache.org/jira/browse/SPARK-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940185#comment-15940185 ]
Takeshi Yamamuro commented on SPARK-8586:
-----------------------------------------
This issue still happens in v2.1.0 and the master.
{code}
scala> sql("add jar sql/hive/src/test/resources/hive-hcatalog-core-0.13.1.jar")
scala> sql("drop table if exists jsonTable")
scala> sql("CREATE TABLE jsonTable(key int, val string) ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'")
scala> spark.range(100).selectExpr("CAST(id AS INT) AS key", "'xxx' AS val").write.insertInto("jsonTable")
scala> sql("SELECT * FROM jsonTable").show
java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hive.hcatalog.data.JsonSerDe
at org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializerClass(TableDesc.java:74)
at org.apache.spark.sql.hive.execution.HiveTableScanExec.addColumnMetadataToConf(HiveTableScanExec.scala:114)
at org.apache.spark.sql.hive.execution.HiveTableScanExec.<init>(HiveTableScanExec.scala:94)
at org.apache.spark.sql.hive.HiveStrategies$HiveTableScans$$anonfun$11.apply(HiveStrategies.scala:207)
at org.apache.spark.sql.hive.HiveStrategies$HiveTableScans$$anonfun$11.apply(HiveStrategies.scala:207)
at org.apache.spark.sql.execution.SparkPlanner.pruneFilterProject(SparkPlanner.scala:92)
at org.apache.spark.sql.hive.HiveStrategies$HiveTableScans$.apply(HiveStrategies.scala:203)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:62)
at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:62)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
{code}
> SQL add jar command does not work well with Scala REPL
> ------------------------------------------------------
>
> Key: SPARK-8586
> URL: https://issues.apache.org/jira/browse/SPARK-8586
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.4.0, 2.1.0
> Reporter: Yin Huai
> Assignee: Yin Huai
> Priority: Critical
>
> Seems SparkIMain always resets the context class loader in {{loadAndRunReq}}. So, SerDe added through add jar command may not be loaded in the context class loader when we lookup the table.
> For example, the following code will fail when we try to show the table.
> {code}
> hive.sql("add jar sql/hive/src/test/resources/hive-hcatalog-core-0.13.1.jar")
> hive.sql("drop table if exists jsonTable")
> hive.sql("CREATE TABLE jsonTable(key int, val string) ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'")
> hive.createDataFrame((1 to 100).map(i => (i, s"str$i"))).toDF("key", "val").insertInto("jsonTable")
> hive.table("jsonTable").show
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org