You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Michael Munday (Jira)" <ji...@apache.org> on 2020/09/21 12:45:00 UTC
[jira] [Updated] (SPARK-32952) Test failure on IBM Z:
CoalesceShufflePartitionsSuite: - determining the number of reducers:
complex query 1
[ https://issues.apache.org/jira/browse/SPARK-32952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Munday updated SPARK-32952:
-----------------------------------
Environment: Linux on IBM Z (s390x). (was: *{color:#FF0000}{color}*Linux on IBM Z (s390x).)
> Test failure on IBM Z: CoalesceShufflePartitionsSuite: - determining the number of reducers: complex query 1
> ------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-32952
> URL: https://issues.apache.org/jira/browse/SPARK-32952
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.1
> Environment: Linux on IBM Z (s390x).
> Reporter: Michael Munday
> Priority: Minor
> Labels: big-endian
>
> I'm seeing the test 'CoalesceShufflePartitionsSuite: - determining the number of reducers: complex query 1' fail on IBM Z with the wrong number of partitions (1 instead of 2). It's strange because none of the other tests fail.
> I'd be grateful for any hints as to how the number of partitions is calculated. Could that calculation be affected by incorrect unsafe code? Is there a way to trace the calculation?
> {noformat}
> CoalesceShufflePartitionsSuite:
> - determining the number of reducers: aggregate operator(minNumPostShufflePartitions: 5)
> - determining the number of reducers: join operator(minNumPostShufflePartitions: 5)
> - determining the number of reducers: complex query 1(minNumPostShufflePartitions: 5)
> - determining the number of reducers: complex query 2(minNumPostShufflePartitions: 5)
> - determining the number of reducers: plan already partitioned(minNumPostShufflePartitions: 5)
> - determining the number of reducers: aggregate operator
> - determining the number of reducers: join operator
> - determining the number of reducers: complex query 1 *** FAILED ***
> 1 did not equal 2 (CoalesceShufflePartitionsSuite.scala:221)
> - determining the number of reducers: complex query 2
> - determining the number of reducers: plan already partitioned
> - SPARK-24705 adaptive query execution works correctly when exchange reuse enabled
> - Do not reduce the number of shuffle partition for repartition
> - Union two datasets with different pre-shuffle partition number{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org