You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "zuotingbing (Jira)" <ji...@apache.org> on 2025/12/27 06:39:00 UTC
[jira] [Updated] (SPARK-54855) FileNotFoundException on _temporary directory
[ https://issues.apache.org/jira/browse/SPARK-54855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
zuotingbing updated SPARK-54855:
--------------------------------
Description:
My use case like this:
spark-sql> CREATE TABLE test_parquet (a INT, b INT, c INT, d INT) USING PARQUET PARTITIONED BY (b, c);
INSERT overwrite TABLE test_parquet PARTITION(b=1, c=1) SELECT 1,2; // app1
INSERT overwrite TABLE test_parquet PARTITION(b=2, c=2) SELECT 3,4; // app2
Two apps are writing to the same table but different partitions simultaneously. Then, when they commited job/task, they both cleans up $dest/_temporary dir by Hadoop FileOutputCommitter and they cause the problem. We definitely want to do better for this case.
> FileNotFoundException on _temporary directory
> ---------------------------------------------
>
> Key: SPARK-54855
> URL: https://issues.apache.org/jira/browse/SPARK-54855
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.5.6
> Reporter: zuotingbing
> Priority: Major
>
> My use case like this:
> spark-sql> CREATE TABLE test_parquet (a INT, b INT, c INT, d INT) USING PARQUET PARTITIONED BY (b, c);
> INSERT overwrite TABLE test_parquet PARTITION(b=1, c=1) SELECT 1,2; // app1
> INSERT overwrite TABLE test_parquet PARTITION(b=2, c=2) SELECT 3,4; // app2
> Two apps are writing to the same table but different partitions simultaneously. Then, when they commited job/task, they both cleans up $dest/_temporary dir by Hadoop FileOutputCommitter and they cause the problem. We definitely want to do better for this case.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org