You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2021/06/20 03:28:00 UTC
[jira] [Commented] (SPARK-35804) can't read external hive table on
spark
[ https://issues.apache.org/jira/browse/SPARK-35804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17366088#comment-17366088 ]
Hyukjin Kwon commented on SPARK-35804:
--------------------------------------
Spark 2.3 and 2.4 are EOL. Can you try it in Spark 3.0+?
> can't read external hive table on spark
> ---------------------------------------
>
> Key: SPARK-35804
> URL: https://issues.apache.org/jira/browse/SPARK-35804
> Project: Spark
> Issue Type: Bug
> Components: PySpark, Spark Core, Spark Shell
> Affects Versions: 2.3.2
> Environment: hdp 3.1.4
> hive-hcatalog-core-3.1.0.3.1.4.0-315.jar & hive-hcatalog-core-3.1.2 both I've tried
>
> Reporter: cao zhiyu
> Priority: Major
> Labels: JSON, external-tables, hive, spark
>
> I create a external hive table with hdfs file which is formatted as json string.
> I can read the data field of this hive table with the help of org.apache.hive.hcatalog.data.JsonSerDe which is packed in hive-hcatalog-core.jar in hive shell.
> But when I try to use the spark (pyspark ,spark-shell or whatever) ,I just can't read it.
> It gave me a error Table: Unable to get field from serde: org.apache.hive.hcatalog.data.JsonSerDe
> I've copied the jar (hive-hcatalog-core.jar) to $spark_home/jars and yarn libs and rerun ,there is no effect,even use --jars $jar_path/hive-hcatalog-core.jar.But when I browse the webpage of spark task ,I can actually find the jar in the env list.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org