You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by ul...@apache.org on 2026/07/10 08:42:25 UTC

(kyuubi) branch master updated: [KYUUBI #7551] [Spark] Optimize InferRebalanceAndSortOrders to support skip non-cheap keys and local sort

This is an automated email from the ASF dual-hosted git repository.

ulysses-you pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kyuubi.git


The following commit(s) were added to refs/heads/master by this push:
     new 0c6f752754 [KYUUBI #7551] [Spark] Optimize InferRebalanceAndSortOrders to support skip non-cheap keys and local sort
0c6f752754 is described below

commit 0c6f752754e600b4eb1f715c5b1dee05c22825f4
Author: Xiduo You <ul...@apache.org>
AuthorDate: Fri Jul 10 16:42:13 2026 +0800

    [KYUUBI #7551] [Spark] Optimize InferRebalanceAndSortOrders to support skip non-cheap keys and local sort
    
    ### Why are the changes needed?
    
    The `InferRebalanceAndSortOrders` optimization infers rebalance partitioning
    and sort columns from the original query (e.g. join keys) to improve the
    compression ratio before writing. In practice this can regress performance in
    two cases:
    
    1. **Expensive inferred columns.** When the inferred rebalance/sort keys are
       non-trivial expressions, evaluating them during the extra shuffle and local
       sort adds CPU cost that can outweigh the compression benefit.
    2. **Unwanted local sort.** Some workloads only want the file-layout benefit
       from rebalance and do not want the additional local `Sort` before writing.
    
    This PR makes the inference tunable so users can opt out of expensive-column
    inference and skip the sort while keeping the rebalance.
    
    Two new configs are added (default behavior is unchanged except that inference
    now restricts to cheap columns by default):
    
    | Config | Default | Description |
    |---|---|---|
    | `spark.sql.optimizer.inferRebalanceAndSortOrdersWithCheapColumns.enabled` | `true` | Only infer rebalance/sort columns when all inferred columns are cheap expressions (attributes, foldable values, or field extractions over cheap expressions). |
    | `spark.sql.optimizer.skipInferRebalanceAndSortOrders.enabled` | `false` | Only infer the rebalance partition columns and skip inferring the sort orders. |
    
    Both configs only take effect when
    `spark.sql.optimizer.inferRebalanceAndSortOrders.enabled` is `true`.
    
    Implementation notes:
    
    - `InferRebalanceAndSortOrders.infer` gains an `onlyInferWithCheapColumns`
      parameter. A new `isCheap` helper classifies an expression as cheap when it is
      an `Attribute` / `OuterReference` / `BoundReference`, foldable, or an `Alias` /
      `ExtractValue` whose children are all cheap. When the flag is set and any
      inferred column is not cheap, inference returns `None` and the rule falls back
      to a plain rebalance.
    - `RebalanceBeforeWritingBase.buildRebalance` reads the two new configs and
      skips the local `Sort` when `skipInferRebalanceAndSortOrders.enabled` is on.
    - The change is applied consistently across the Spark `4.1`, `4.0`, and `3.5`
      extension modules, and the rule docs table is updated.
    
    ### How was this patch tested?
    
    - Added `Skip inferring sort orders` — end-to-end check that enabling
      `skipInferRebalanceAndSortOrders.enabled` drops the inferred local `Sort`
      while keeping the `RebalancePartitions`.
    - Added `Infer rebalance and sort orders only with cheap columns` — unit-level
      check on `InferRebalanceAndSortOrders.infer`: expensive join keys
      (`col1 + 1`) are not inferred when the cheap-column restriction is on and are
      inferred when it is off; cheap attribute keys are inferred regardless.
    - Tests added to the `RebalanceBeforeWritingSuite` of all three modules.
    
    Ran locally (all green):
    
    ```
    build/mvn -Pspark-4.1 -Pscala-2.13 test -pl extensions/spark/kyuubi-extension-spark-4-1 -am
      -Dtest=none -DwildcardSuites=org.apache.spark.sql.RebalanceBeforeWritingSuite   # 12/12
    build/mvn -Pspark-4.0 -Pscala-2.13 test -pl extensions/spark/kyuubi-extension-spark-4-0 -am
      -Dtest=none -DwildcardSuites=org.apache.spark.sql.RebalanceBeforeWritingSuite   # 12/12
    build/mvn -Pspark-3.5 test -pl extensions/spark/kyuubi-extension-spark-3-5 -am
      -Dtest=none -DwildcardSuites=org.apache.spark.sql.RebalanceBeforeWritingSuite   # 14/14
    ```
    
    `dev/reformat` run; no style changes required.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Yes.
    
    Assisted-by: Claude Opus 4.8
    
    Closes #7551 from ulysses-you/infer.
    
    Closes #7551
    
    0b6003bef [Xiduo You] comment
    54320bcdd [Xiduo You] Optimize InferRebalanceAndSortOrders to support skip non-cheap keys and local sort
    
    Authored-by: Xiduo You <ul...@apache.org>
    Signed-off-by: Xiduo You <ul...@apache.org>
---
 docs/extensions/engines/spark/rules.md             |  2 +
 .../kyuubi/sql/InferRebalanceAndSortOrders.scala   | 24 +++++-
 .../org/apache/kyuubi/sql/KyuubiSQLConf.scala      | 19 +++++
 .../apache/kyuubi/sql/RebalanceBeforeWriting.scala |  7 +-
 .../spark/sql/RebalanceBeforeWritingSuite.scala    | 87 +++++++++++++++++++++-
 .../kyuubi/sql/InferRebalanceAndSortOrders.scala   | 24 +++++-
 .../org/apache/kyuubi/sql/KyuubiSQLConf.scala      | 19 +++++
 .../apache/kyuubi/sql/RebalanceBeforeWriting.scala |  7 +-
 .../spark/sql/RebalanceBeforeWritingSuite.scala    | 87 +++++++++++++++++++++-
 .../kyuubi/sql/InferRebalanceAndSortOrders.scala   | 24 +++++-
 .../org/apache/kyuubi/sql/KyuubiSQLConf.scala      | 19 +++++
 .../apache/kyuubi/sql/RebalanceBeforeWriting.scala |  7 +-
 .../spark/sql/RebalanceBeforeWritingSuite.scala    | 87 +++++++++++++++++++++-
 13 files changed, 395 insertions(+), 18 deletions(-)

diff --git a/docs/extensions/engines/spark/rules.md b/docs/extensions/engines/spark/rules.md
index 94e0a797f2..3999336c9d 100644
--- a/docs/extensions/engines/spark/rules.md
+++ b/docs/extensions/engines/spark/rules.md
@@ -95,6 +95,8 @@ Kyuubi provides some configs to make these feature easy to use.
 | spark.sql.optimizer.zorderUsingOriginalOrdering.enabled             | false                                  | When true and `spark.sql.optimizer.rebalanceBeforeZorder.enabled` is true, we do sort by the original ordering i.e. lexicographical order.                                                                                                                                                                                                           | 1.6.0  |
 | spark.sql.optimizer.inferRebalanceAndSortOrders.enabled             | false                                  | When ture, infer columns for rebalance and sort orders from original query, e.g. the join keys from join. It can avoid compression ratio regression.                                                                                                                                                                                                 | 1.7.0  |
 | spark.sql.optimizer.inferRebalanceAndSortOrdersMaxColumns           | 3                                      | The max columns of inferred columns.                                                                                                                                                                                                                                                                                                                 | 1.7.0  |
+| spark.sql.optimizer.inferRebalanceAndSortOrders.cheapColumnsOnly    | true                                   | When true and `spark.sql.optimizer.inferRebalanceAndSortOrders.enabled` is true, only infer the rebalance and sort columns when all inferred columns are cheap expressions, i.e. attributes, foldable values, or field extractions over cheap expressions. This avoids evaluating expensive expressions during the inferred shuffle and sort.        | 1.12.0 |
+| spark.sql.optimizer.inferRebalanceAndSortOrders.skipSort            | false                                  | When true and `spark.sql.optimizer.inferRebalanceAndSortOrders.enabled` is true, only infer the rebalance partition columns and skip inferring the sort orders. Skipping the sort avoids a local sort before writing when only the file layout from rebalance is wanted.                                                                             | 1.12.0 |
 | spark.sql.optimizer.insertRepartitionBeforeWriteIfNoShuffle.enabled | false                                  | When true, add repartition even if the original plan does not have shuffle.                                                                                                                                                                                                                                                                          | 1.7.0  |
 | spark.sql.optimizer.finalStageConfigIsolationWriteOnly.enabled      | true                                   | When true, only enable final stage isolation for writing.                                                                                                                                                                                                                                                                                            | 1.7.0  |
 | spark.sql.adaptive.rebalancePartitionsAdvisoryPartitionSizeInBytes  | none                                   | The advisory partition size for the `RebalancePartitions` operator injected by `insertRepartitionBeforeWrite` and related rules. When set, it is passed to `optAdvisoryPartitionSize` of the `RebalancePartitions` node. It takes precedence over `spark.sql.finalStage.adaptive.advisoryPartitionSizeInBytes`.                                      | 1.12.0 |
diff --git a/extensions/spark/kyuubi-extension-spark-3-5/src/main/scala/org/apache/kyuubi/sql/InferRebalanceAndSortOrders.scala b/extensions/spark/kyuubi-extension-spark-3-5/src/main/scala/org/apache/kyuubi/sql/InferRebalanceAndSortOrders.scala
index 3b840f2a01..ba80eda078 100644
--- a/extensions/spark/kyuubi-extension-spark-3-5/src/main/scala/org/apache/kyuubi/sql/InferRebalanceAndSortOrders.scala
+++ b/extensions/spark/kyuubi-extension-spark-3-5/src/main/scala/org/apache/kyuubi/sql/InferRebalanceAndSortOrders.scala
@@ -19,7 +19,7 @@ package org.apache.kyuubi.sql
 
 import scala.annotation.tailrec
 
-import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute, AttributeSet, Expression, NamedExpression, UnaryExpression}
+import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute, AttributeSet, BoundReference, Expression, ExtractValue, NamedExpression, OuterReference, UnaryExpression}
 import org.apache.spark.sql.catalyst.planning.ExtractEquiJoinKeys
 import org.apache.spark.sql.catalyst.plans.{FullOuter, Inner, LeftAnti, LeftOuter, LeftSemi, RightOuter}
 import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Filter, Generate, LogicalPlan, Project, Sort, SubqueryAlias, View, Window}
@@ -53,7 +53,16 @@ object InferRebalanceAndSortOrders {
     }.toMap
   }
 
-  def infer(plan: LogicalPlan): Option[PartitioningAndOrdering] = {
+  def isCheap(e: Expression): Boolean = e match {
+    case _: Attribute | _: OuterReference | _: BoundReference => true
+    case _ if e.foldable => true
+    case _: Alias | _: ExtractValue => e.children.forall(isCheap)
+    case _ => false
+  }
+
+  def infer(
+      plan: LogicalPlan,
+      onlyInferWithCheapColumns: Boolean): Option[PartitioningAndOrdering] = {
     def candidateKeys(
         input: LogicalPlan,
         output: AttributeSet = AttributeSet.empty): Option[PartitioningAndOrdering] = {
@@ -107,10 +116,19 @@ object InferRebalanceAndSortOrders {
       }
     }
 
-    candidateKeys(plan).map { case (partitioning, ordering) =>
+    val partitioningAndSort = candidateKeys(plan).map { case (partitioning, ordering) =>
       (
         partitioning.filter(_.references.subsetOf(plan.outputSet)),
         ordering.filter(_.references.subsetOf(plan.outputSet)))
     }
+    val allCheap = partitioningAndSort.exists {
+      case (partitionings, sorts) =>
+        partitionings.forall(isCheap) && sorts.forall(isCheap)
+    }
+    if (!onlyInferWithCheapColumns || allCheap) {
+      partitioningAndSort
+    } else {
+      None
+    }
   }
 }
diff --git a/extensions/spark/kyuubi-extension-spark-3-5/src/main/scala/org/apache/kyuubi/sql/KyuubiSQLConf.scala b/extensions/spark/kyuubi-extension-spark-3-5/src/main/scala/org/apache/kyuubi/sql/KyuubiSQLConf.scala
index d68c5fb7f9..481ffdf116 100644
--- a/extensions/spark/kyuubi-extension-spark-3-5/src/main/scala/org/apache/kyuubi/sql/KyuubiSQLConf.scala
+++ b/extensions/spark/kyuubi-extension-spark-3-5/src/main/scala/org/apache/kyuubi/sql/KyuubiSQLConf.scala
@@ -143,6 +143,25 @@ object KyuubiSQLConf {
       .booleanConf
       .createWithDefault(false)
 
+  val SKIP_INFER_SORT_ORDERS =
+    buildConf("spark.sql.optimizer.inferRebalanceAndSortOrders.skipSort")
+      .doc(s"When true and `${INFER_REBALANCE_AND_SORT_ORDERS.key}` is true, only infer the " +
+        s"rebalance partition columns and skip inferring the sort orders. Skipping the sort " +
+        s"avoids a local sort before writing when only the file layout from rebalance is wanted.")
+      .version("1.12.0")
+      .booleanConf
+      .createWithDefault(false)
+
+  val INFER_REBALANCE_AND_SORT_ORDERS_WITH_CHEAP_COLUMNS =
+    buildConf("spark.sql.optimizer.inferRebalanceAndSortOrders.cheapColumnsOnly")
+      .doc(s"When true and `${INFER_REBALANCE_AND_SORT_ORDERS.key}` is true, only infer the " +
+        s"rebalance and sort columns when all inferred columns are cheap expressions, i.e. " +
+        s"attributes, foldable values, or field extractions over cheap expressions. This avoids " +
+        s"evaluating expensive expressions during the inferred shuffle and sort.")
+      .version("1.12.0")
+      .booleanConf
+      .createWithDefault(true)
+
   val INFER_REBALANCE_AND_SORT_ORDERS_MAX_COLUMNS =
     buildConf("spark.sql.optimizer.inferRebalanceAndSortOrdersMaxColumns")
       .doc("The max columns of inferred columns.")
diff --git a/extensions/spark/kyuubi-extension-spark-3-5/src/main/scala/org/apache/kyuubi/sql/RebalanceBeforeWriting.scala b/extensions/spark/kyuubi-extension-spark-3-5/src/main/scala/org/apache/kyuubi/sql/RebalanceBeforeWriting.scala
index b74b33cc6f..79aaa5c48e 100644
--- a/extensions/spark/kyuubi-extension-spark-3-5/src/main/scala/org/apache/kyuubi/sql/RebalanceBeforeWriting.scala
+++ b/extensions/spark/kyuubi-extension-spark-3-5/src/main/scala/org/apache/kyuubi/sql/RebalanceBeforeWriting.scala
@@ -77,14 +77,17 @@ trait RebalanceBeforeWritingBase extends Rule[LogicalPlan] {
         optAdvisoryPartitionSize = advisoryPartitionSize)
     } else {
       val maxColumns = conf.getConf(KyuubiSQLConf.INFER_REBALANCE_AND_SORT_ORDERS_MAX_COLUMNS)
-      val inferred = InferRebalanceAndSortOrders.infer(query)
+      val onlyInferWithCheapColumns = conf.getConf(
+        KyuubiSQLConf.INFER_REBALANCE_AND_SORT_ORDERS_WITH_CHEAP_COLUMNS)
+      val inferred = InferRebalanceAndSortOrders.infer(query, onlyInferWithCheapColumns)
       if (inferred.isDefined) {
         val (partitioning, ordering) = inferred.get
         val rebalance = RebalancePartitions(
           partitioning.take(maxColumns),
           query,
           optAdvisoryPartitionSize = advisoryPartitionSize)
-        if (ordering.nonEmpty) {
+        val skipInferOrder = conf.getConf(KyuubiSQLConf.SKIP_INFER_SORT_ORDERS)
+        if (!skipInferOrder && ordering.nonEmpty) {
           val sortOrders = ordering.take(maxColumns).map(o => SortOrder(o, Ascending))
           Sort(sortOrders, false, rebalance)
         } else {
diff --git a/extensions/spark/kyuubi-extension-spark-3-5/src/test/scala/org/apache/spark/sql/RebalanceBeforeWritingSuite.scala b/extensions/spark/kyuubi-extension-spark-3-5/src/test/scala/org/apache/spark/sql/RebalanceBeforeWritingSuite.scala
index 11563292fb..d543633a7d 100644
--- a/extensions/spark/kyuubi-extension-spark-3-5/src/test/scala/org/apache/spark/sql/RebalanceBeforeWritingSuite.scala
+++ b/extensions/spark/kyuubi-extension-spark-3-5/src/test/scala/org/apache/spark/sql/RebalanceBeforeWritingSuite.scala
@@ -25,7 +25,7 @@ import org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationComm
 import org.apache.spark.sql.hive.HiveUtils
 import org.apache.spark.sql.hive.execution.{InsertIntoHiveDirCommand, InsertIntoHiveTable}
 
-import org.apache.kyuubi.sql.KyuubiSQLConf
+import org.apache.kyuubi.sql.{InferRebalanceAndSortOrders, KyuubiSQLConf}
 
 class RebalanceBeforeWritingSuite extends KyuubiSparkSQLExtensionTest {
 
@@ -305,6 +305,91 @@ class RebalanceBeforeWritingSuite extends KyuubiSparkSQLExtensionTest {
     }
   }
 
+  test("Skip inferring sort orders") {
+    def checkShuffleAndSort(dataWritingCommand: LogicalPlan, sSize: Int, rSize: Int): Unit = {
+      assert(dataWritingCommand.isInstanceOf[DataWritingCommand])
+      val plan = dataWritingCommand.asInstanceOf[DataWritingCommand].query
+      assert(plan.collect { case s: Sort => s }.size == sSize)
+      assert(plan.collect {
+        case r: RebalancePartitions if r.partitionExpressions.size == rSize => r
+      }.nonEmpty || rSize == 0)
+    }
+
+    withTable("t", "input1", "input2") {
+      sql(s"CREATE TABLE t (c1 int, c2 long) USING PARQUET PARTITIONED BY (p string)")
+      sql(s"CREATE TABLE input1 USING PARQUET AS SELECT * FROM VALUES(1,2),(1,3)")
+      sql(s"CREATE TABLE input2 USING PARQUET AS SELECT * FROM VALUES(1,3),(1,3)")
+
+      val insert =
+        s"""
+           |INSERT INTO TABLE t PARTITION(p='a')
+           |SELECT /*+ broadcast(input2) */ input1.col1, input2.col1
+           |FROM input1
+           |JOIN input2
+           |ON input1.col1 = input2.col1
+           |""".stripMargin
+
+      // when skip is disabled (default), both rebalance and sort orders are inferred
+      withSQLConf(
+        KyuubiSQLConf.INFER_REBALANCE_AND_SORT_ORDERS.key -> "true",
+        KyuubiSQLConf.SKIP_INFER_SORT_ORDERS.key -> "false") {
+        checkShuffleAndSort(sql(insert).queryExecution.analyzed, 1, 1)
+      }
+
+      // when skip is enabled, only the rebalance is inferred and the sort is skipped
+      withSQLConf(
+        KyuubiSQLConf.INFER_REBALANCE_AND_SORT_ORDERS.key -> "true",
+        KyuubiSQLConf.SKIP_INFER_SORT_ORDERS.key -> "true") {
+        checkShuffleAndSort(sql(insert).queryExecution.analyzed, 0, 1)
+      }
+    }
+  }
+
+  test("Infer rebalance and sort orders only with cheap columns") {
+    withTable("input1", "input2") {
+      sql(s"CREATE TABLE input1 USING PARQUET AS SELECT * FROM VALUES(1,2),(1,3)")
+      sql(s"CREATE TABLE input2 USING PARQUET AS SELECT * FROM VALUES(1,3),(1,3)")
+
+      // the join keys `input1.col1 + 1` / `input2.col1 + 1` are not cheap expressions
+      val expensivePlan = sql(
+        s"""
+           |SELECT input1.col1, input2.col1
+           |FROM input1
+           |JOIN input2
+           |ON input1.col1 + 1 = input2.col1 + 1
+           |""".stripMargin).queryExecution.analyzed
+
+      // when only cheap columns are allowed, the expensive keys are not inferred
+      assert(InferRebalanceAndSortOrders.infer(expensivePlan, true).isEmpty)
+      // when the cheap-column restriction is disabled, the expensive keys are inferred
+      InferRebalanceAndSortOrders.infer(expensivePlan, false) match {
+        case Some((partitioning, ordering)) =>
+          assert(partitioning.nonEmpty)
+          assert(ordering.nonEmpty)
+        case None => fail("expected inferred columns when cheap-column restriction is disabled")
+      }
+
+      // the join keys `input1.col1` / `input2.col1` are cheap attributes
+      val cheapPlan = sql(
+        s"""
+           |SELECT input1.col1, input2.col1
+           |FROM input1
+           |JOIN input2
+           |ON input1.col1 = input2.col1
+           |""".stripMargin).queryExecution.analyzed
+
+      // cheap columns are inferred regardless of the restriction
+      Seq(true, false).foreach { onlyCheap =>
+        InferRebalanceAndSortOrders.infer(cheapPlan, onlyCheap) match {
+          case Some((partitioning, ordering)) =>
+            assert(partitioning.nonEmpty)
+            assert(ordering.nonEmpty)
+          case None => fail(s"expected inferred columns when onlyCheap=$onlyCheap")
+        }
+      }
+    }
+  }
+
   test("Test rebalance in InsertIntoHiveDirCommand") {
     withSQLConf(
       HiveUtils.CONVERT_METASTORE_PARQUET.key -> "false",
diff --git a/extensions/spark/kyuubi-extension-spark-4-0/src/main/scala/org/apache/kyuubi/sql/InferRebalanceAndSortOrders.scala b/extensions/spark/kyuubi-extension-spark-4-0/src/main/scala/org/apache/kyuubi/sql/InferRebalanceAndSortOrders.scala
index 3b840f2a01..ba80eda078 100644
--- a/extensions/spark/kyuubi-extension-spark-4-0/src/main/scala/org/apache/kyuubi/sql/InferRebalanceAndSortOrders.scala
+++ b/extensions/spark/kyuubi-extension-spark-4-0/src/main/scala/org/apache/kyuubi/sql/InferRebalanceAndSortOrders.scala
@@ -19,7 +19,7 @@ package org.apache.kyuubi.sql
 
 import scala.annotation.tailrec
 
-import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute, AttributeSet, Expression, NamedExpression, UnaryExpression}
+import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute, AttributeSet, BoundReference, Expression, ExtractValue, NamedExpression, OuterReference, UnaryExpression}
 import org.apache.spark.sql.catalyst.planning.ExtractEquiJoinKeys
 import org.apache.spark.sql.catalyst.plans.{FullOuter, Inner, LeftAnti, LeftOuter, LeftSemi, RightOuter}
 import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Filter, Generate, LogicalPlan, Project, Sort, SubqueryAlias, View, Window}
@@ -53,7 +53,16 @@ object InferRebalanceAndSortOrders {
     }.toMap
   }
 
-  def infer(plan: LogicalPlan): Option[PartitioningAndOrdering] = {
+  def isCheap(e: Expression): Boolean = e match {
+    case _: Attribute | _: OuterReference | _: BoundReference => true
+    case _ if e.foldable => true
+    case _: Alias | _: ExtractValue => e.children.forall(isCheap)
+    case _ => false
+  }
+
+  def infer(
+      plan: LogicalPlan,
+      onlyInferWithCheapColumns: Boolean): Option[PartitioningAndOrdering] = {
     def candidateKeys(
         input: LogicalPlan,
         output: AttributeSet = AttributeSet.empty): Option[PartitioningAndOrdering] = {
@@ -107,10 +116,19 @@ object InferRebalanceAndSortOrders {
       }
     }
 
-    candidateKeys(plan).map { case (partitioning, ordering) =>
+    val partitioningAndSort = candidateKeys(plan).map { case (partitioning, ordering) =>
       (
         partitioning.filter(_.references.subsetOf(plan.outputSet)),
         ordering.filter(_.references.subsetOf(plan.outputSet)))
     }
+    val allCheap = partitioningAndSort.exists {
+      case (partitionings, sorts) =>
+        partitionings.forall(isCheap) && sorts.forall(isCheap)
+    }
+    if (!onlyInferWithCheapColumns || allCheap) {
+      partitioningAndSort
+    } else {
+      None
+    }
   }
 }
diff --git a/extensions/spark/kyuubi-extension-spark-4-0/src/main/scala/org/apache/kyuubi/sql/KyuubiSQLConf.scala b/extensions/spark/kyuubi-extension-spark-4-0/src/main/scala/org/apache/kyuubi/sql/KyuubiSQLConf.scala
index d68c5fb7f9..481ffdf116 100644
--- a/extensions/spark/kyuubi-extension-spark-4-0/src/main/scala/org/apache/kyuubi/sql/KyuubiSQLConf.scala
+++ b/extensions/spark/kyuubi-extension-spark-4-0/src/main/scala/org/apache/kyuubi/sql/KyuubiSQLConf.scala
@@ -143,6 +143,25 @@ object KyuubiSQLConf {
       .booleanConf
       .createWithDefault(false)
 
+  val SKIP_INFER_SORT_ORDERS =
+    buildConf("spark.sql.optimizer.inferRebalanceAndSortOrders.skipSort")
+      .doc(s"When true and `${INFER_REBALANCE_AND_SORT_ORDERS.key}` is true, only infer the " +
+        s"rebalance partition columns and skip inferring the sort orders. Skipping the sort " +
+        s"avoids a local sort before writing when only the file layout from rebalance is wanted.")
+      .version("1.12.0")
+      .booleanConf
+      .createWithDefault(false)
+
+  val INFER_REBALANCE_AND_SORT_ORDERS_WITH_CHEAP_COLUMNS =
+    buildConf("spark.sql.optimizer.inferRebalanceAndSortOrders.cheapColumnsOnly")
+      .doc(s"When true and `${INFER_REBALANCE_AND_SORT_ORDERS.key}` is true, only infer the " +
+        s"rebalance and sort columns when all inferred columns are cheap expressions, i.e. " +
+        s"attributes, foldable values, or field extractions over cheap expressions. This avoids " +
+        s"evaluating expensive expressions during the inferred shuffle and sort.")
+      .version("1.12.0")
+      .booleanConf
+      .createWithDefault(true)
+
   val INFER_REBALANCE_AND_SORT_ORDERS_MAX_COLUMNS =
     buildConf("spark.sql.optimizer.inferRebalanceAndSortOrdersMaxColumns")
       .doc("The max columns of inferred columns.")
diff --git a/extensions/spark/kyuubi-extension-spark-4-0/src/main/scala/org/apache/kyuubi/sql/RebalanceBeforeWriting.scala b/extensions/spark/kyuubi-extension-spark-4-0/src/main/scala/org/apache/kyuubi/sql/RebalanceBeforeWriting.scala
index a55766f51f..5a8c43d969 100644
--- a/extensions/spark/kyuubi-extension-spark-4-0/src/main/scala/org/apache/kyuubi/sql/RebalanceBeforeWriting.scala
+++ b/extensions/spark/kyuubi-extension-spark-4-0/src/main/scala/org/apache/kyuubi/sql/RebalanceBeforeWriting.scala
@@ -76,14 +76,17 @@ trait RebalanceBeforeWritingBase extends Rule[LogicalPlan] {
         optAdvisoryPartitionSize = advisoryPartitionSize)
     } else {
       val maxColumns = conf.getConf(KyuubiSQLConf.INFER_REBALANCE_AND_SORT_ORDERS_MAX_COLUMNS)
-      val inferred = InferRebalanceAndSortOrders.infer(query)
+      val onlyInferWithCheapColumns = conf.getConf(
+        KyuubiSQLConf.INFER_REBALANCE_AND_SORT_ORDERS_WITH_CHEAP_COLUMNS)
+      val inferred = InferRebalanceAndSortOrders.infer(query, onlyInferWithCheapColumns)
       if (inferred.isDefined) {
         val (partitioning, ordering) = inferred.get
         val rebalance = RebalancePartitions(
           partitioning.take(maxColumns),
           query,
           optAdvisoryPartitionSize = advisoryPartitionSize)
-        if (ordering.nonEmpty) {
+        val skipInferOrder = conf.getConf(KyuubiSQLConf.SKIP_INFER_SORT_ORDERS)
+        if (!skipInferOrder && ordering.nonEmpty) {
           val sortOrders = ordering.take(maxColumns).map(o => SortOrder(o, Ascending))
           Sort(sortOrders, false, rebalance)
         } else {
diff --git a/extensions/spark/kyuubi-extension-spark-4-0/src/test/scala/org/apache/spark/sql/RebalanceBeforeWritingSuite.scala b/extensions/spark/kyuubi-extension-spark-4-0/src/test/scala/org/apache/spark/sql/RebalanceBeforeWritingSuite.scala
index de8e463fbd..8a413b298f 100644
--- a/extensions/spark/kyuubi-extension-spark-4-0/src/test/scala/org/apache/spark/sql/RebalanceBeforeWritingSuite.scala
+++ b/extensions/spark/kyuubi-extension-spark-4-0/src/test/scala/org/apache/spark/sql/RebalanceBeforeWritingSuite.scala
@@ -24,7 +24,7 @@ import org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationComm
 import org.apache.spark.sql.hive.HiveUtils
 import org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
-import org.apache.kyuubi.sql.KyuubiSQLConf
+import org.apache.kyuubi.sql.{InferRebalanceAndSortOrders, KyuubiSQLConf}
 
 class RebalanceBeforeWritingSuite extends KyuubiSparkSQLExtensionTest {
 
@@ -304,6 +304,91 @@ class RebalanceBeforeWritingSuite extends KyuubiSparkSQLExtensionTest {
     }
   }
 
+  test("Skip inferring sort orders") {
+    def checkShuffleAndSort(dataWritingCommand: LogicalPlan, sSize: Int, rSize: Int): Unit = {
+      assert(dataWritingCommand.isInstanceOf[DataWritingCommand])
+      val plan = dataWritingCommand.asInstanceOf[DataWritingCommand].query
+      assert(plan.collect { case s: Sort => s }.size == sSize)
+      assert(plan.collect {
+        case r: RebalancePartitions if r.partitionExpressions.size == rSize => r
+      }.nonEmpty || rSize == 0)
+    }
+
+    withTable("t", "input1", "input2") {
+      sql(s"CREATE TABLE t (c1 int, c2 long) USING PARQUET PARTITIONED BY (p string)")
+      sql(s"CREATE TABLE input1 USING PARQUET AS SELECT * FROM VALUES(1,2),(1,3)")
+      sql(s"CREATE TABLE input2 USING PARQUET AS SELECT * FROM VALUES(1,3),(1,3)")
+
+      val insert =
+        s"""
+           |INSERT INTO TABLE t PARTITION(p='a')
+           |SELECT /*+ broadcast(input2) */ input1.col1, input2.col1
+           |FROM input1
+           |JOIN input2
+           |ON input1.col1 = input2.col1
+           |""".stripMargin
+
+      // when skip is disabled (default), both rebalance and sort orders are inferred
+      withSQLConf(
+        KyuubiSQLConf.INFER_REBALANCE_AND_SORT_ORDERS.key -> "true",
+        KyuubiSQLConf.SKIP_INFER_SORT_ORDERS.key -> "false") {
+        checkShuffleAndSort(sql(insert).queryExecution.analyzed, 1, 1)
+      }
+
+      // when skip is enabled, only the rebalance is inferred and the sort is skipped
+      withSQLConf(
+        KyuubiSQLConf.INFER_REBALANCE_AND_SORT_ORDERS.key -> "true",
+        KyuubiSQLConf.SKIP_INFER_SORT_ORDERS.key -> "true") {
+        checkShuffleAndSort(sql(insert).queryExecution.analyzed, 0, 1)
+      }
+    }
+  }
+
+  test("Infer rebalance and sort orders only with cheap columns") {
+    withTable("input1", "input2") {
+      sql(s"CREATE TABLE input1 USING PARQUET AS SELECT * FROM VALUES(1,2),(1,3)")
+      sql(s"CREATE TABLE input2 USING PARQUET AS SELECT * FROM VALUES(1,3),(1,3)")
+
+      // the join keys `input1.col1 + 1` / `input2.col1 + 1` are not cheap expressions
+      val expensivePlan = sql(
+        s"""
+           |SELECT input1.col1, input2.col1
+           |FROM input1
+           |JOIN input2
+           |ON input1.col1 + 1 = input2.col1 + 1
+           |""".stripMargin).queryExecution.analyzed
+
+      // when only cheap columns are allowed, the expensive keys are not inferred
+      assert(InferRebalanceAndSortOrders.infer(expensivePlan, true).isEmpty)
+      // when the cheap-column restriction is disabled, the expensive keys are inferred
+      InferRebalanceAndSortOrders.infer(expensivePlan, false) match {
+        case Some((partitioning, ordering)) =>
+          assert(partitioning.nonEmpty)
+          assert(ordering.nonEmpty)
+        case None => fail("expected inferred columns when cheap-column restriction is disabled")
+      }
+
+      // the join keys `input1.col1` / `input2.col1` are cheap attributes
+      val cheapPlan = sql(
+        s"""
+           |SELECT input1.col1, input2.col1
+           |FROM input1
+           |JOIN input2
+           |ON input1.col1 = input2.col1
+           |""".stripMargin).queryExecution.analyzed
+
+      // cheap columns are inferred regardless of the restriction
+      Seq(true, false).foreach { onlyCheap =>
+        InferRebalanceAndSortOrders.infer(cheapPlan, onlyCheap) match {
+          case Some((partitioning, ordering)) =>
+            assert(partitioning.nonEmpty)
+            assert(ordering.nonEmpty)
+          case None => fail(s"expected inferred columns when onlyCheap=$onlyCheap")
+        }
+      }
+    }
+  }
+
   test("advisory partition size - finalStage config takes precedence") {
     withSQLConf(
       KyuubiSQLConf.INSERT_REPARTITION_BEFORE_WRITE.key -> "true",
diff --git a/extensions/spark/kyuubi-extension-spark-4-1/src/main/scala/org/apache/kyuubi/sql/InferRebalanceAndSortOrders.scala b/extensions/spark/kyuubi-extension-spark-4-1/src/main/scala/org/apache/kyuubi/sql/InferRebalanceAndSortOrders.scala
index 3b840f2a01..ba80eda078 100644
--- a/extensions/spark/kyuubi-extension-spark-4-1/src/main/scala/org/apache/kyuubi/sql/InferRebalanceAndSortOrders.scala
+++ b/extensions/spark/kyuubi-extension-spark-4-1/src/main/scala/org/apache/kyuubi/sql/InferRebalanceAndSortOrders.scala
@@ -19,7 +19,7 @@ package org.apache.kyuubi.sql
 
 import scala.annotation.tailrec
 
-import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute, AttributeSet, Expression, NamedExpression, UnaryExpression}
+import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute, AttributeSet, BoundReference, Expression, ExtractValue, NamedExpression, OuterReference, UnaryExpression}
 import org.apache.spark.sql.catalyst.planning.ExtractEquiJoinKeys
 import org.apache.spark.sql.catalyst.plans.{FullOuter, Inner, LeftAnti, LeftOuter, LeftSemi, RightOuter}
 import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Filter, Generate, LogicalPlan, Project, Sort, SubqueryAlias, View, Window}
@@ -53,7 +53,16 @@ object InferRebalanceAndSortOrders {
     }.toMap
   }
 
-  def infer(plan: LogicalPlan): Option[PartitioningAndOrdering] = {
+  def isCheap(e: Expression): Boolean = e match {
+    case _: Attribute | _: OuterReference | _: BoundReference => true
+    case _ if e.foldable => true
+    case _: Alias | _: ExtractValue => e.children.forall(isCheap)
+    case _ => false
+  }
+
+  def infer(
+      plan: LogicalPlan,
+      onlyInferWithCheapColumns: Boolean): Option[PartitioningAndOrdering] = {
     def candidateKeys(
         input: LogicalPlan,
         output: AttributeSet = AttributeSet.empty): Option[PartitioningAndOrdering] = {
@@ -107,10 +116,19 @@ object InferRebalanceAndSortOrders {
       }
     }
 
-    candidateKeys(plan).map { case (partitioning, ordering) =>
+    val partitioningAndSort = candidateKeys(plan).map { case (partitioning, ordering) =>
       (
         partitioning.filter(_.references.subsetOf(plan.outputSet)),
         ordering.filter(_.references.subsetOf(plan.outputSet)))
     }
+    val allCheap = partitioningAndSort.exists {
+      case (partitionings, sorts) =>
+        partitionings.forall(isCheap) && sorts.forall(isCheap)
+    }
+    if (!onlyInferWithCheapColumns || allCheap) {
+      partitioningAndSort
+    } else {
+      None
+    }
   }
 }
diff --git a/extensions/spark/kyuubi-extension-spark-4-1/src/main/scala/org/apache/kyuubi/sql/KyuubiSQLConf.scala b/extensions/spark/kyuubi-extension-spark-4-1/src/main/scala/org/apache/kyuubi/sql/KyuubiSQLConf.scala
index d68c5fb7f9..481ffdf116 100644
--- a/extensions/spark/kyuubi-extension-spark-4-1/src/main/scala/org/apache/kyuubi/sql/KyuubiSQLConf.scala
+++ b/extensions/spark/kyuubi-extension-spark-4-1/src/main/scala/org/apache/kyuubi/sql/KyuubiSQLConf.scala
@@ -143,6 +143,25 @@ object KyuubiSQLConf {
       .booleanConf
       .createWithDefault(false)
 
+  val SKIP_INFER_SORT_ORDERS =
+    buildConf("spark.sql.optimizer.inferRebalanceAndSortOrders.skipSort")
+      .doc(s"When true and `${INFER_REBALANCE_AND_SORT_ORDERS.key}` is true, only infer the " +
+        s"rebalance partition columns and skip inferring the sort orders. Skipping the sort " +
+        s"avoids a local sort before writing when only the file layout from rebalance is wanted.")
+      .version("1.12.0")
+      .booleanConf
+      .createWithDefault(false)
+
+  val INFER_REBALANCE_AND_SORT_ORDERS_WITH_CHEAP_COLUMNS =
+    buildConf("spark.sql.optimizer.inferRebalanceAndSortOrders.cheapColumnsOnly")
+      .doc(s"When true and `${INFER_REBALANCE_AND_SORT_ORDERS.key}` is true, only infer the " +
+        s"rebalance and sort columns when all inferred columns are cheap expressions, i.e. " +
+        s"attributes, foldable values, or field extractions over cheap expressions. This avoids " +
+        s"evaluating expensive expressions during the inferred shuffle and sort.")
+      .version("1.12.0")
+      .booleanConf
+      .createWithDefault(true)
+
   val INFER_REBALANCE_AND_SORT_ORDERS_MAX_COLUMNS =
     buildConf("spark.sql.optimizer.inferRebalanceAndSortOrdersMaxColumns")
       .doc("The max columns of inferred columns.")
diff --git a/extensions/spark/kyuubi-extension-spark-4-1/src/main/scala/org/apache/kyuubi/sql/RebalanceBeforeWriting.scala b/extensions/spark/kyuubi-extension-spark-4-1/src/main/scala/org/apache/kyuubi/sql/RebalanceBeforeWriting.scala
index a55766f51f..5a8c43d969 100644
--- a/extensions/spark/kyuubi-extension-spark-4-1/src/main/scala/org/apache/kyuubi/sql/RebalanceBeforeWriting.scala
+++ b/extensions/spark/kyuubi-extension-spark-4-1/src/main/scala/org/apache/kyuubi/sql/RebalanceBeforeWriting.scala
@@ -76,14 +76,17 @@ trait RebalanceBeforeWritingBase extends Rule[LogicalPlan] {
         optAdvisoryPartitionSize = advisoryPartitionSize)
     } else {
       val maxColumns = conf.getConf(KyuubiSQLConf.INFER_REBALANCE_AND_SORT_ORDERS_MAX_COLUMNS)
-      val inferred = InferRebalanceAndSortOrders.infer(query)
+      val onlyInferWithCheapColumns = conf.getConf(
+        KyuubiSQLConf.INFER_REBALANCE_AND_SORT_ORDERS_WITH_CHEAP_COLUMNS)
+      val inferred = InferRebalanceAndSortOrders.infer(query, onlyInferWithCheapColumns)
       if (inferred.isDefined) {
         val (partitioning, ordering) = inferred.get
         val rebalance = RebalancePartitions(
           partitioning.take(maxColumns),
           query,
           optAdvisoryPartitionSize = advisoryPartitionSize)
-        if (ordering.nonEmpty) {
+        val skipInferOrder = conf.getConf(KyuubiSQLConf.SKIP_INFER_SORT_ORDERS)
+        if (!skipInferOrder && ordering.nonEmpty) {
           val sortOrders = ordering.take(maxColumns).map(o => SortOrder(o, Ascending))
           Sort(sortOrders, false, rebalance)
         } else {
diff --git a/extensions/spark/kyuubi-extension-spark-4-1/src/test/scala/org/apache/spark/sql/RebalanceBeforeWritingSuite.scala b/extensions/spark/kyuubi-extension-spark-4-1/src/test/scala/org/apache/spark/sql/RebalanceBeforeWritingSuite.scala
index de8e463fbd..8a413b298f 100644
--- a/extensions/spark/kyuubi-extension-spark-4-1/src/test/scala/org/apache/spark/sql/RebalanceBeforeWritingSuite.scala
+++ b/extensions/spark/kyuubi-extension-spark-4-1/src/test/scala/org/apache/spark/sql/RebalanceBeforeWritingSuite.scala
@@ -24,7 +24,7 @@ import org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationComm
 import org.apache.spark.sql.hive.HiveUtils
 import org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
-import org.apache.kyuubi.sql.KyuubiSQLConf
+import org.apache.kyuubi.sql.{InferRebalanceAndSortOrders, KyuubiSQLConf}
 
 class RebalanceBeforeWritingSuite extends KyuubiSparkSQLExtensionTest {
 
@@ -304,6 +304,91 @@ class RebalanceBeforeWritingSuite extends KyuubiSparkSQLExtensionTest {
     }
   }
 
+  test("Skip inferring sort orders") {
+    def checkShuffleAndSort(dataWritingCommand: LogicalPlan, sSize: Int, rSize: Int): Unit = {
+      assert(dataWritingCommand.isInstanceOf[DataWritingCommand])
+      val plan = dataWritingCommand.asInstanceOf[DataWritingCommand].query
+      assert(plan.collect { case s: Sort => s }.size == sSize)
+      assert(plan.collect {
+        case r: RebalancePartitions if r.partitionExpressions.size == rSize => r
+      }.nonEmpty || rSize == 0)
+    }
+
+    withTable("t", "input1", "input2") {
+      sql(s"CREATE TABLE t (c1 int, c2 long) USING PARQUET PARTITIONED BY (p string)")
+      sql(s"CREATE TABLE input1 USING PARQUET AS SELECT * FROM VALUES(1,2),(1,3)")
+      sql(s"CREATE TABLE input2 USING PARQUET AS SELECT * FROM VALUES(1,3),(1,3)")
+
+      val insert =
+        s"""
+           |INSERT INTO TABLE t PARTITION(p='a')
+           |SELECT /*+ broadcast(input2) */ input1.col1, input2.col1
+           |FROM input1
+           |JOIN input2
+           |ON input1.col1 = input2.col1
+           |""".stripMargin
+
+      // when skip is disabled (default), both rebalance and sort orders are inferred
+      withSQLConf(
+        KyuubiSQLConf.INFER_REBALANCE_AND_SORT_ORDERS.key -> "true",
+        KyuubiSQLConf.SKIP_INFER_SORT_ORDERS.key -> "false") {
+        checkShuffleAndSort(sql(insert).queryExecution.analyzed, 1, 1)
+      }
+
+      // when skip is enabled, only the rebalance is inferred and the sort is skipped
+      withSQLConf(
+        KyuubiSQLConf.INFER_REBALANCE_AND_SORT_ORDERS.key -> "true",
+        KyuubiSQLConf.SKIP_INFER_SORT_ORDERS.key -> "true") {
+        checkShuffleAndSort(sql(insert).queryExecution.analyzed, 0, 1)
+      }
+    }
+  }
+
+  test("Infer rebalance and sort orders only with cheap columns") {
+    withTable("input1", "input2") {
+      sql(s"CREATE TABLE input1 USING PARQUET AS SELECT * FROM VALUES(1,2),(1,3)")
+      sql(s"CREATE TABLE input2 USING PARQUET AS SELECT * FROM VALUES(1,3),(1,3)")
+
+      // the join keys `input1.col1 + 1` / `input2.col1 + 1` are not cheap expressions
+      val expensivePlan = sql(
+        s"""
+           |SELECT input1.col1, input2.col1
+           |FROM input1
+           |JOIN input2
+           |ON input1.col1 + 1 = input2.col1 + 1
+           |""".stripMargin).queryExecution.analyzed
+
+      // when only cheap columns are allowed, the expensive keys are not inferred
+      assert(InferRebalanceAndSortOrders.infer(expensivePlan, true).isEmpty)
+      // when the cheap-column restriction is disabled, the expensive keys are inferred
+      InferRebalanceAndSortOrders.infer(expensivePlan, false) match {
+        case Some((partitioning, ordering)) =>
+          assert(partitioning.nonEmpty)
+          assert(ordering.nonEmpty)
+        case None => fail("expected inferred columns when cheap-column restriction is disabled")
+      }
+
+      // the join keys `input1.col1` / `input2.col1` are cheap attributes
+      val cheapPlan = sql(
+        s"""
+           |SELECT input1.col1, input2.col1
+           |FROM input1
+           |JOIN input2
+           |ON input1.col1 = input2.col1
+           |""".stripMargin).queryExecution.analyzed
+
+      // cheap columns are inferred regardless of the restriction
+      Seq(true, false).foreach { onlyCheap =>
+        InferRebalanceAndSortOrders.infer(cheapPlan, onlyCheap) match {
+          case Some((partitioning, ordering)) =>
+            assert(partitioning.nonEmpty)
+            assert(ordering.nonEmpty)
+          case None => fail(s"expected inferred columns when onlyCheap=$onlyCheap")
+        }
+      }
+    }
+  }
+
   test("advisory partition size - finalStage config takes precedence") {
     withSQLConf(
       KyuubiSQLConf.INSERT_REPARTITION_BEFORE_WRITE.key -> "true",