You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "yy (JIRA)" <ji...@apache.org> on 2018/09/07 08:50:00 UTC
[jira] [Created] (SPARK-25367) Hive table created by Spark
dataFrame has incompatiable schema in spark and hive
yy created SPARK-25367:
--------------------------
Summary: Hive table created by Spark dataFrame has incompatiable schema in spark and hive
Key: SPARK-25367
URL: https://issues.apache.org/jira/browse/SPARK-25367
Project: Spark
Issue Type: Bug
Components: Spark Shell
Affects Versions: 2.2.1
Environment: spark2.2.1
hive1.2.1
spark conf: hive-site.xml configured the metastore information, which is the same as in hive.
Reporter: yy
We save the created dataframe object as a hive table in orc/parquet format in the spark shell.
After we modified the column type (int to double) of this table in hive jdbc, we found the column type queried in spark-shell didn't change, but changed in hive jdbc. After we restarted the spark-shell, this table's column type is still incompatible as showed in hive jdbc.
The coding process are as follows:
spark-shell:
val df = spark.read.json("examples/src/main/resources/people.json");
df.write.format("orc").saveAsTable("people_test");
spark.catalog.refreshTable("people_test")
spark.sql("desc people").show()
hive:
alter table people_test change column age age1 double;
desc people_test;
spark-shell:
spark.sql("desc people").show()
We also tested in spark-shell by creating a table using spark.sql("create table XXX()"), and the modified columns also changed in spark.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org