You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "yaooqinn (via GitHub)" <gi...@apache.org> on 2025/11/05 07:23:05 UTC

[PR] [SPARK-54193][SHUFFLE] Deprecated spark.shuffle.server.finalizeShuffleMergeThreadsPercent [spark]

yaooqinn opened a new pull request, #52893:
URL: https://github.com/apache/spark/pull/52893

   ### What changes were proposed in this pull request?
   
   Deprecated spark.shuffle.server.finalizeShuffleMergeThreadsPercent first, and then we can remove finalizeWorker usages in the future.
   
   ### Why are the changes needed?
   In the Netty project, those related APIs are
   - removed from the main branch https://github.com/netty/netty/pull/8778
   - and deprecated in 4.2 branch https://github.com/netty/netty/pull/14538
   
   ### Does this PR introduce _any_ user-facing change?
   no
   
   ### How was this patch tested?
   
   I verified this by a SparkPi local test
   ```scala
   bin/run-example -c spark.shuffle.server.finalizeShuffleMergeThreadsPercent=10 SparkPi
   WARNING: Using incubator modules: jdk.incubator.vector
   Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
   25/11/05 10:59:14 WARN SparkConf: The configuration key 'spark.shuffle.server.finalizeShuffleMergeThreadsPercent' has been deprecated as of Spark 4.2.0 and may be removed in the future. Using separate finalizeWorkers could be problematic due to the underlying netty layer
   ```
   
   ### Was this patch authored or co-authored using generative AI tooling?
   no
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54193][SHUFFLE] Deprecated spark.shuffle.server.finalizeShuffleMergeThreadsPercent [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #52893: [SPARK-54193][SHUFFLE] Deprecated spark.shuffle.server.finalizeShuffleMergeThreadsPercent
URL: https://github.com/apache/spark/pull/52893


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54193][SHUFFLE] Deprecated spark.shuffle.server.finalizeShuffleMergeThreadsPercent [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #52893:
URL: https://github.com/apache/spark/pull/52893#issuecomment-3623249677

   To @mridulm , `Netty` community decided the functionality like `addLast` is unstable for the production due to the race condition. AFAIK, it has several issues before in `Netty` layer although we didn't get the error reports in Apache community yet because this is fairly new feature of Apache Spark 4.0.0, [SPARK-43987](https://issues.apache.org/jira/browse/SPARK-43987)
   
   https://github.com/apache/spark/blob/6bfb83315e4596037790020a48d0747c308a317d/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ShuffleTransportContext.java#L106
   
   I believe it's good to inform the users this unstable API dependency before getting this spread out further.
   
   Maybe, we had better deprecated this at Apache Spark 4.1.0. WDTY, @mridulm ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54193][SHUFFLE] Deprecated spark.shuffle.server.finalizeShuffleMergeThreadsPercent [spark]

Posted by "mridulm (via GitHub)" <gi...@apache.org>.
mridulm commented on PR #52893:
URL: https://github.com/apache/spark/pull/52893#issuecomment-3619806026

   @yaooqinn , @dongjoon-hyun I did not quite understand the rationale for deprecating these configs.
   The functionality which they provide is required for production stability - is the deprecation simply due to netty master not supporting these ?
   
   If yes, what is the current alternative proposed in netty for existing usages ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54193][SHUFFLE] Deprecated spark.shuffle.server.finalizeShuffleMergeThreadsPercent [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #52893:
URL: https://github.com/apache/spark/pull/52893#issuecomment-3623259118

   The bottom line is that we don't want to be locked in due to this new unstable dependency of SPARK-43987.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54193][SHUFFLE] Deprecated spark.shuffle.server.finalizeShuffleMergeThreadsPercent [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #52893:
URL: https://github.com/apache/spark/pull/52893#issuecomment-3491984833

   Merged to master for Apache Spark 4.2.0.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-54193][SHUFFLE] Deprecated spark.shuffle.server.finalizeShuffleMergeThreadsPercent [spark]

Posted by "mridulm (via GitHub)" <gi...@apache.org>.
mridulm commented on PR #52893:
URL: https://github.com/apache/spark/pull/52893#issuecomment-3626150257

   I was not proposing locking into unstable dependency- rather explore alternatives so that we can continue to support features required for production stability of Apache Spark.
   
   From these PR's, it appeared that deprecation of the config was only due to corresponding netty api getting deprecated/removed.
   
   There are alternatives discussed in the netty PR - do they not work for us ? (I have not investigated - so please do point me in right direction if I am missing it !)
   
   If they do, do we need to deprecate these configs ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org