You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yuming Wang (Jira)" <ji...@apache.org> on 2023/03/30 05:45:00 UTC
[jira] [Commented] (SPARK-42932) Spark 3.3.2, with hadoop3, Error with java.io.IOException: Mkdirs failed to create file
[ https://issues.apache.org/jira/browse/SPARK-42932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706702#comment-17706702 ]
Yuming Wang commented on SPARK-42932:
-------------------------------------
I can't reproduce this issue. Maybe it's your disk issue?
> Spark 3.3.2, with hadoop3, Error with java.io.IOException: Mkdirs failed to create file
> ----------------------------------------------------------------------------------------
>
> Key: SPARK-42932
> URL: https://issues.apache.org/jira/browse/SPARK-42932
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, Spark Submit
> Affects Versions: 3.3.2
> Reporter: shamim
> Priority: Major
>
> We are using spark 3.3.2 with hadoop 3 coming with spark.
> [https://www.apache.org/dyn/closer.lua/spark/spark-3.3.2/spark-3.3.2-bin-hadoop3.tgz]
> [https://www.apache.org/dyn/closer.lua/spark/spark-3.3.2/spark-3.3.2-bin-hadoop2.tgz]
> Spark in our application is used as standalone , and we are not using HDFS file system.
> Spark is writing on local file system.
> Same spark version 3.3.2 is working fine with hadoop 2. but with hadoop 3 , we are getting below issue.
>
> 23/03/18 20:23:24 WARN TaskSetManager: Lost task 4.0 in stage 0.0 (TID 4) (10.64.109.72 executor 0): java.io.IOException: Mkdirs failed to create [file:/var/backup/_temporary/0/_temporary/attempt_202301182023173234741341853025716_0005_m_000004_0|file://var/backup/_temporary/0/_temporary/attempt_202301182023173234741341853025716_0005_m_000004_0] (exists=false, cwd=[file:/opt/spark-3.3.2/work/app-20230118202317-0001/0|file://opt/spark-3.3.0/work/app-20230118202317-0001/0])
> at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:515)
> at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:500)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1195)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1081)
> at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:113)
> at org.apache.spark.internal.io.HadoopMapRedWriteConfigUtil.initWriter(SparkHadoopWriter.scala:238)
> at org.apache.spark.internal.io.SparkHadoopWriter$.executeTask(SparkHadoopWriter.scala:126)
> at org.apache.spark.internal.io.SparkHadoopWriter$.$anonfun$write$1(SparkHadoopWriter.scala:88)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
> at org.apache.spark.scheduler.Task.run(Task.scala:136)
> at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org