You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Lee Sysuke <sy...@gmail.com> on 2023/03/17 08:41:58 UTC
Divided by zero occurs in PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse
Hi all,
Recently I have a SQL job run with push based shuffle, and met exception as below. It seems that there’s no element in the bitmaps which stores merge chunk meta. See org.apache.spark.storage.PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse.
I would like to know is it a bug that we should not createChunkBlockInfos when bitmaps is empty or the bitmaps should never be empty here ?
Caused by: java.lang.ArithmeticException: / by zero
at org.apache.spark.storage.PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse(PushBasedFetchHelper.scala:117)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:980)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:84)
at org.apache.spark.util.CompletionIterator.next(CompletionIterator.scala:29)
at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:31)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage13.sort_addToSorter_0$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage13.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
Re: Divided by zero occurs in PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse
Posted by Cheng Pan <ch...@apache.org>.
It‘s SPARK-37675[1], which has been fixed in 3.3.0, 3.2.2.
BTW, better to attach more information when report issue, e.g. versions of
Spark, JDK, Hadoop, etc.
[1] https://issues.apache.org/jira/browse/SPARK-37675
Thanks,
Cheng Pan
On Mar 17, 2023 at 16:41:58, Lee Sysuke <sy...@gmail.com> wrote:
> Hi all,
>
> Recently I have a SQL job run with push based shuffle, and met exception
> as below. It seems that there’s no element in the bitmaps which stores
> merge chunk meta. See
> org.apache.spark.storage.PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse.
>
> I would like to know is it a bug that we should not createChunkBlockInfos
> when bitmaps is empty or the bitmaps should never be empty here ?
>
>
> Caused by: java.lang.ArithmeticException: / by zero
> at
> org.apache.spark.storage.PushBasedFetchHelper.createChunkBlockInfosFromMetaResponse(PushBasedFetchHelper.scala:117)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:980)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:84)
> at
> org.apache.spark.util.CompletionIterator.next(CompletionIterator.scala:29)
> at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492)
> at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:31)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
> at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage13.sort_addToSorter_0$(Unknown
> Source)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage13.processNext(Unknown
> Source)
> at
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
>
>
>
>