Failed to shuffle write

[yguang11 <gu...@gmail.com>]

Hello,
I am fairly new to Spark, recently I was debugging some Spark application
failures, one issue I found is that the executor failed to write with the
following stack trace:
2018-09-23 05:05:38 ERROR Executor:91 - Exception in task 1037.0 in stage
14.0 (TID 33041)
 java.io.FileNotFoundException:
/mnt/yarn/usercache/svc_di_payments/appcache/application_1533840530908_0650/blockmgr-774f7b90-cd87-4589-b4cf-6baea80ac58c/36/shuffle_10_1037_0.data.3b42eb09-8a67-4b8d-990e-0e03417b1fe3
(No such file or directory)
     at java.io.FileOutputStream.open0(Native Method)
     at java.io.FileOutputStream.open(FileOutputStream.java:270)
     at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
     at
org.apache.spark.storage.DiskBlockObjectWriter.initialize(DiskBlockObjectWriter.scala:103)
     at
org.apache.spark.storage.DiskBlockObjectWriter.open(DiskBlockObjectWriter.scala:116)
     at
org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:237)
     at
org.apache.spark.util.collection.WritablePartitionedPairCollection$$anon$1.writeNext(WritablePartitionedPairCollection.scala:56)
     at
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:699)
     at
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:72)
     at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
     at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
     at org.apache.spark.scheduler.Task.run(Task.scala:109)
     at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
     at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
     at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
     at java.lang.Thread.run(Thread.java:748)

What looks strange in this case, is that the container is running
Executor:54 (lots of logs for that executor), however, for the failures it
says Executor:91, is that normal?

Thanks,
Guang



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org