[jira] [Created] (SPARK-56588) Dynamic partition overwrite (partitionOverwriteMode=DYNAMIC) behaves as append for Spark 4.1+ and HDFS

["Kazuyuki Tanimura (Jira)" <ji...@apache.org>]

Kazuyuki Tanimura created SPARK-56588:
-----------------------------------------

             Summary: Dynamic partition overwrite (partitionOverwriteMode=DYNAMIC) behaves as append for Spark 4.1+ and HDFS 
                 Key: SPARK-56588
                 URL: https://issues.apache.org/jira/browse/SPARK-56588
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 4.1.1, 4.2.0, 4.1.2
            Reporter: Kazuyuki Tanimura


spark.sql.sources.partitionOverwriteMode=DYNAMIC does not work as expected for Spark  4.1+ and HDFS. Partition overwrites behave as appends, old data is preserved alongside new data in the same partition directory, resulting in duplicate rows. This affects both the session config path (spark.conf.set()) and the inline write option path (df.write.option())

This looks related to https://issues.apache.org/jira/browse/SPARK-54248

Setting 

 —conf spark.sql.parquet.output.committer.class=org.apache.parquet.hadoop.ParquetOutputCommitter —conf spark.sql.sources.commitProtocolClass=org.apache.spark.sql.execution.datasources.SQLHadoopMapReduceCommitProtocol 

Solves the problem.

Spark automatically applies org.apache.spark.internal.io.cloud.BindingParquetOutputCommitter / org.apache.spark.internal.io.cloud.PathOutputCommitProtocol if the confs are empty and spark-hadoop-cloud module is available. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org