[jira] [Resolved] (SPARK-34091) Shuffle batch fetch can't be disabled once it's enabled previously

["Wenchen Fan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-34091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wenchen Fan resolved SPARK-34091.
---------------------------------
    Fix Version/s: 3.1.1
       Resolution: Fixed

Issue resolved by pull request 31155
[https://github.com/apache/spark/pull/31155]

> Shuffle batch fetch can't be disabled once it's enabled previously
> ------------------------------------------------------------------
>
>                 Key: SPARK-34091
>                 URL: https://issues.apache.org/jira/browse/SPARK-34091
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 3.0.0, 3.1.0
>            Reporter: wuyi
>            Assignee: wuyi
>            Priority: Major
>             Fix For: 3.1.1
>
>
> {code:java}
>   if (SQLConf.get.fetchShuffleBlocksInBatch) {
>     dependency.rdd.context.setLocalProperty(
>       SortShuffleManager.FETCH_SHUFFLE_BLOCKS_IN_BATCH_ENABLED_KEY, "true")
>   }
> {code}
> The current code has a problem that once we set `fetchShuffleBlocksInBatch` to true first, we can never disable the batch fetch even if set `fetchShuffleBlocksInBatch` to false later.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org