You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "ulysses-you (via GitHub)" <gi...@apache.org> on 2026/07/22 12:29:48 UTC

[PR] [SPARK] Use the larger advisory partition size when both configs are set [kyuubi]

ulysses-you opened a new pull request, #7581:
URL: https://github.com/apache/kyuubi/pull/7581

   ### Why are the changes needed?
   
   When both `spark.sql.adaptive.rebalancePartitionsAdvisoryPartitionSizeInBytes`
   and `spark.sql.finalStage.adaptive.advisoryPartitionSizeInBytes` are set,
   `KyuubiSQLConf.getAdvisoryPartitionSize` previously let the rebalance config
   unconditionally take precedence. This could pick a smaller advisory partition
   size than the finalStage config even when the finalStage value is larger.
   
   This changes the resolution so the larger of the two values is used when both
   are set, which better reflects the intent of both knobs. The doc for the
   rebalance config is updated accordingly. The change is applied to the
   spark-3.5, 4.0 and 4.1 extension modules.
   
   ### How was this patch tested?
   
   Updated `RebalanceBeforeWritingSuite` in all three modules to cover:
   - both set, rebalance larger -> rebalance wins
   - both set, finalStage larger -> finalStage wins
   - both set to the same value
   
   Ran the suite locally:
   
   ```
   build/mvn test -pl extensions/spark/kyuubi-extension-spark-3-5 -am -Pspark-3.5 \
     -Dtest=none -DwildcardSuites=org.apache.spark.sql.RebalanceBeforeWritingSuite   # 14 passed
   build/mvn test -pl extensions/spark/kyuubi-extension-spark-4-0 -am -Pspark-4.0 -Pscala-2.13 \
     -Dtest=none -DwildcardSuites=org.apache.spark.sql.RebalanceBeforeWritingSuite   # 12 passed
   build/mvn test -pl extensions/spark/kyuubi-extension-spark-4-1 -am -Pspark-4.1 -Pscala-2.13 \
     -Dtest=none -DwildcardSuites=org.apache.spark.sql.RebalanceBeforeWritingSuite   # 12 passed
   ```
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Yes.
   
   Assisted-by: Claude Opus 4.8


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [SPARK] Use the larger advisory partition size when both configs are set [kyuubi]

Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you closed pull request #7581: [SPARK] Use the larger advisory partition size when both configs are set
URL: https://github.com/apache/kyuubi/pull/7581


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [SPARK] Use the larger advisory partition size when both configs are set [kyuubi]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #7581:
URL: https://github.com/apache/kyuubi/pull/7581#issuecomment-5045794376

   4.2 is missing?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [SPARK] Use the larger advisory partition size when both configs are set [kyuubi]

Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on PR #7581:
URL: https://github.com/apache/kyuubi/pull/7581#issuecomment-5054542862

   thanks, merging to master


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org