[jira] [Created] (SPARK-27555) cannot create table by using the hive default fileformat in both hive-site.xml and spark-defaults.conf

["Hui WANG (JIRA)" <ji...@apache.org>]

Hui WANG created SPARK-27555:
--------------------------------

             Summary: cannot create table by using the hive default fileformat in both hive-site.xml and spark-defaults.conf
                 Key: SPARK-27555
                 URL: https://issues.apache.org/jira/browse/SPARK-27555
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.3.2
            Reporter: Hui WANG


I already seen https://issues.apache.org/jira/browse/SPARK-17620 

and https://issues.apache.org/jira/browse/SPARK-18397

and I check source code of Spark for the change of  set "spark.sql.hive.covertCTAS=true" and then spark will use "spark.sql.sources.default" which is parquet as storage format in "create table as select" scenario.

But my case is just create table without select. When I set  hive.default.fileformat=parquet in hive-site.xml or set  spark.hadoop.hive.default.fileformat=parquet in spark-defaults.conf, after create a table, when i check the hive table, it still use textfile fileformat.

 

HiveSerDe gets the value of the hive.default.fileformat parameter from SQLConf

The parameter values in SQLConf are copied from SparkContext's SparkConf at SparkSession initialization, while the configuration parameters in hive-site.xml are loaded into SparkContext's hadoopConfiguration parameters by SharedState, And all the config with "spark.hadoop" conf are setted to hadoopconfig, so the configuration does not take effect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org