You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Wenchen Fan (Jira)" <ji...@apache.org> on 2020/10/15 07:03:00 UTC
[jira] [Resolved] (SPARK-32932) AQE local shuffle reader breaks
repartitioning for dynamic partition overwrite
[ https://issues.apache.org/jira/browse/SPARK-32932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenchen Fan resolved SPARK-32932.
---------------------------------
Fix Version/s: 3.1.0
Resolution: Fixed
> AQE local shuffle reader breaks repartitioning for dynamic partition overwrite
> ------------------------------------------------------------------------------
>
> Key: SPARK-32932
> URL: https://issues.apache.org/jira/browse/SPARK-32932
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Manu Zhang
> Assignee: Manu Zhang
> Priority: Major
> Fix For: 3.1.0
>
>
> With AQE, local shuffle reader breaks users' repartitioning for dynamic partition overwrite as in the following case.
> {code:java}
> test("repartition with local reader") {
> withSQLConf(SQLConf.PARTITION_OVERWRITE_MODE.key -> PartitionOverwriteMode.DYNAMIC.toString,
> SQLConf.SHUFFLE_PARTITIONS.key -> "5",
> SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "true") {
> withTable("t") {
> val data = for (
> i <- 1 to 10;
> j <- 1 to 3
> ) yield (i, j)
> data.toDF("a", "b")
> .repartition($"b")
> .write
> .partitionBy("b")
> .mode("overwrite")
> .saveAsTable("t")
> assert(spark.read.table("t").inputFiles.length == 3)
> }
> }
> }{code}
> -Coalescing shuffle partitions could also break it.-
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org