You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "zhengshubin (via GitHub)" <gi...@apache.org> on 2024/02/01 08:07:44 UTC
[I] [BUG] spark procedure occur Shuffle data lost for shuffle xx partitionId xx frequently [incubator-celeborn]
zhengshubin opened a new issue, #2275:
URL: https://github.com/apache/incubator-celeborn/issues/2275
### What is the bug(with logs or screenshots)?
A clear and concise description of what the bug is.
celeborn 0.3.2 spark 3.3.3
log:
org.apache.kyuubi.KyuubiSQLException:
Error operating ExecuteStatement: org.apache.spark.SparkException: Job aborted due to stage failure: Task 8 in stage 8.0 failed 4 times,
org.apache.celeborn.common.exception.CelebornIOException:
Shuffle data lost for shuffle 1 partitionId 9!
org.apache.celeborn.client.ShuffleClientImpl.loadFileGroup(ShuffleClientImpl.java:1603)
org.apache.celeborn.client.ShuffleClientImpl.readPartition(ShuffleClientImpl.java:1612)
org.apache.spark.shuffle.celeborn.CelebornShuffleReader.$anonfun$read$1(CelebornShuffleReader.scala:89)
org.apache.spark.shuffle.celeborn.CelebornShuffleReader.$anonfun$read$1$adapted(CelebornShuffleReader.scala:81)
scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486)
scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492)
scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:31)
org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
scala.collection.Iterator.foreach(Iterator.scala:943)
scala.collection.Iterator.foreach$(Iterator.scala:943)
scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
org.apache.paimon.spark.commands.WriteIntoPaimonTable.$anonfun$run$6(WriteIntoPaimonTable.scala:141)
org.apache.spark.sql.execution.MapPartitionsExec.$anonfun$doExecute$3(objects.scala:201)
org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:890)
org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:890)
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
### How to reproduce the bug?
Steps to reproduce the bug.
/cc @who-need-to-know
/assign @who-can-solve-this-bug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [BUG] spark procedure occur Shuffle data lost for shuffle xx partitionId xx frequently [incubator-celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas closed issue #2275: [BUG] spark procedure occur Shuffle data lost for shuffle xx partitionId xx frequently
URL: https://github.com/apache/incubator-celeborn/issues/2275
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [BUG] spark procedure occur Shuffle data lost for shuffle xx partitionId xx frequently [incubator-celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on issue #2275:
URL: https://github.com/apache/incubator-celeborn/issues/2275#issuecomment-1920735202
Hi @zhengshubin, Can you provide more details like your Celeborn cluster's scale and workload info?
This error looks like there is a shuffle lost. Shuffle lost in Celeborn occasionally happens when the Celeborn cluster is on high load.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org