[jira] [Created] (SPARK-25389) INSERT OVERWRITE DIRECTORY STORED AS should not generate files with duplicate fields

["Dongjoon Hyun (JIRA)" <ji...@apache.org>]

Dongjoon Hyun created SPARK-25389:
-------------------------------------

             Summary: INSERT OVERWRITE DIRECTORY STORED AS should not generate files with duplicate fields
                 Key: SPARK-25389
                 URL: https://issues.apache.org/jira/browse/SPARK-25389
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.3.1, 2.3.0
            Reporter: Dongjoon Hyun


Like `INSERT OVERWRITE DIRECTORY USING` syntax, `INSERT OVERWRITE DIRECTORY STORED AS` should not generate files with duplicate fields because Spark cannot read those files.

*INSERT OVERWRITE DIRECTORY USING*
{code}
scala> sql("INSERT OVERWRITE DIRECTORY 'file:///tmp/parquet' USING parquet SELECT 'id', 'id2' id")
18/09/09 22:11:29 ERROR InsertIntoDataSourceDirCommand: Failed to write to directory Some(file:///tmp/parquet)
org.apache.spark.sql.AnalysisException: Found duplicate column(s) when inserting into file:/tmp/parquet: `id`;
{code}

*INSERT OVERWRITE DIRECTORY STORED AS*
{code}
scala> sql("INSERT OVERWRITE DIRECTORY 'file:///tmp/parquet' STORED AS parquet SELECT 'id', 'id2' id")

scala> spark.read.parquet("/tmp/parquet").show
18/09/09 22:09:57 WARN DataSource: Found duplicate column(s) in the data schema and the partition schema: `id`;
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org