You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Vivian Kong (Jira)" <ji...@apache.org> on 2022/08/04 13:03:00 UTC
[jira] [Commented] (SPARK-32952) Test failure on IBM Z: CoalesceShufflePartitionsSuite: - determining the number of reducers: complex query 1
[ https://issues.apache.org/jira/browse/SPARK-32952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17575252#comment-17575252 ]
Vivian Kong commented on SPARK-32952:
-------------------------------------
The test is still failing on Spark v3.3.0 on IBM Z. We appreciate any pointers community can share so we can look into it. Thanks.
> Test failure on IBM Z: CoalesceShufflePartitionsSuite: - determining the number of reducers: complex query 1
> ------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-32952
> URL: https://issues.apache.org/jira/browse/SPARK-32952
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.1
> Environment: Linux on IBM Z (s390x).
> Reporter: Michael Munday
> Priority: Minor
> Labels: big-endian
>
> I'm seeing the test 'CoalesceShufflePartitionsSuite: - determining the number of reducers: complex query 1' fail on IBM Z with the wrong number of partitions (1 instead of 2). It's strange because none of the other tests fail.
> I'd be grateful for any hints as to how the number of partitions is calculated. Could that calculation be affected by incorrect unsafe code? Is there a way to trace the calculation?
> {noformat}
> CoalesceShufflePartitionsSuite:
> - determining the number of reducers: aggregate operator(minNumPostShufflePartitions: 5)
> - determining the number of reducers: join operator(minNumPostShufflePartitions: 5)
> - determining the number of reducers: complex query 1(minNumPostShufflePartitions: 5)
> - determining the number of reducers: complex query 2(minNumPostShufflePartitions: 5)
> - determining the number of reducers: plan already partitioned(minNumPostShufflePartitions: 5)
> - determining the number of reducers: aggregate operator
> - determining the number of reducers: join operator
> - determining the number of reducers: complex query 1 *** FAILED ***
> 1 did not equal 2 (CoalesceShufflePartitionsSuite.scala:221)
> - determining the number of reducers: complex query 2
> - determining the number of reducers: plan already partitioned
> - SPARK-24705 adaptive query execution works correctly when exchange reuse enabled
> - Do not reduce the number of shuffle partition for repartition
> - Union two datasets with different pre-shuffle partition number{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org