You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by yu...@apache.org on 2024/12/05 08:04:43 UTC
(incubator-gluten) branch main updated: [GLUTEN-7143][VL] RAS: Remove experimental flags for RAS (#8154)
This is an automated email from the ASF dual-hosted git repository.
yuanzhou pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git
The following commit(s) were added to refs/heads/main by this push:
new f704f09285 [GLUTEN-7143][VL] RAS: Remove experimental flags for RAS (#8154)
f704f09285 is described below
commit f704f092855d664def191935c03d3c284b97f218
Author: Hongze Zhang <ho...@intel.com>
AuthorDate: Thu Dec 5 16:04:39 2024 +0800
[GLUTEN-7143][VL] RAS: Remove experimental flags for RAS (#8154)
Part of #7143 (not to close it)
RAS is considered production-ready now as all Spark / Gluten UTs, TPC-H, TPC-DS ITs have passed for a period of time. More documentations needed but we can remove the experimental flags to state it's ready for use.
The patch removes the flags from doc and code.
---
docs/Configuration.md | 2 +-
.../gluten/extension/columnar/enumerated/EnumeratedApplier.scala | 7 ++-----
shims/common/src/main/scala/org/apache/gluten/GlutenConfig.scala | 7 ++++---
3 files changed, 7 insertions(+), 9 deletions(-)
diff --git a/docs/Configuration.md b/docs/Configuration.md
index cb8efe802e..e217be45ff 100644
--- a/docs/Configuration.md
+++ b/docs/Configuration.md
@@ -23,7 +23,7 @@ You can add these configurations into spark-defaults.conf to enable or disable t
| spark.shuffle.manager | To turn on Gluten Columnar Shuffle Plugin [...]
| spark.gluten.enabled | Enable Gluten, default is true. Just an experimental property. Recommend to enable/disable Gluten through the setting for `spark.plugins`. [...]
| spark.gluten.memory.isolation | (Experimental) Enable isolated memory mode. If true, Gluten controls the maximum off-heap memory can be used by each task to X, X = executor memory / max task slots. It's recommended to set true if Gluten serves concurrent queries within a single session, since not all memory Gluten allocated is guaranteed to be spillable. In the case, the feature should be enabled to avoid OOM. Note when true, setting spark.memory.storageF [...]
-| spark.gluten.ras.enabled | Experimental: Enables RAS (relation algebra selector) during physical planning to generate more efficient query plan. Note, this feature is still in development and may not bring performance profits. [...]
+| spark.gluten.ras.enabled | Enables RAS (relation algebra selector) during physical planning to generate more efficient query plan. Note, this feature doesn't bring performance profits by default. Try exploring option `spark.gluten.ras.costModel` for advanced usage. [...]
| spark.gluten.sql.columnar.maxBatchSize | Number of rows to be processed in each batch. Default value is 4096. [...]
| spark.gluten.sql.columnar.scanOnly | When enabled, this config will overwrite all other operators' enabling, and only Scan and Filter pushdown will be offloaded to native. [...]
| spark.gluten.sql.columnar.batchscan | Enable or Disable Columnar BatchScan, default is true [...]
diff --git a/gluten-core/src/main/scala/org/apache/gluten/extension/columnar/enumerated/EnumeratedApplier.scala b/gluten-core/src/main/scala/org/apache/gluten/extension/columnar/enumerated/EnumeratedApplier.scala
index ff822d1c0a..04cd70656e 100644
--- a/gluten-core/src/main/scala/org/apache/gluten/extension/columnar/enumerated/EnumeratedApplier.scala
+++ b/gluten-core/src/main/scala/org/apache/gluten/extension/columnar/enumerated/EnumeratedApplier.scala
@@ -21,7 +21,6 @@ import org.apache.gluten.extension.columnar.ColumnarRuleApplier.ColumnarRuleCall
import org.apache.gluten.extension.util.AdaptiveContext
import org.apache.gluten.logging.LogLevelUtil
-import org.apache.spark.annotation.Experimental
import org.apache.spark.internal.Logging
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.catalyst.rules.Rule
@@ -31,11 +30,9 @@ import org.apache.spark.sql.execution.SparkPlan
* Columnar rule applier that optimizes, implements Spark plan into Gluten plan by enumerating on
* all the possibilities of executable Gluten plans, then choose the best plan among them.
*
- * NOTE: This is still working in progress. We still have a bunch of heuristic rules in this
- * implementation's rule list. Future work will include removing them from the list then
- * implementing them in EnumeratedTransform.
+ * NOTE: We still have a bunch of heuristic rules in this implementation's rule list. Future work
+ * will include removing them from the list then implementing them in EnumeratedTransform.
*/
-@Experimental
class EnumeratedApplier(
session: SparkSession,
ruleBuilders: Seq[ColumnarRuleCall => Rule[SparkPlan]])
diff --git a/shims/common/src/main/scala/org/apache/gluten/GlutenConfig.scala b/shims/common/src/main/scala/org/apache/gluten/GlutenConfig.scala
index 9ae4c0ce90..4f243f03fb 100644
--- a/shims/common/src/main/scala/org/apache/gluten/GlutenConfig.scala
+++ b/shims/common/src/main/scala/org/apache/gluten/GlutenConfig.scala
@@ -1416,9 +1416,10 @@ object GlutenConfig {
val RAS_ENABLED =
buildConf("spark.gluten.ras.enabled")
.doc(
- "Experimental: Enables RAS (relational algebra selector) during physical " +
- "planning to generate more efficient query plan. Note, this feature is still in " +
- "development and may not bring performance profits.")
+ "Enables RAS (relational algebra selector) during physical " +
+ "planning to generate more efficient query plan. Note, this feature doesn't bring " +
+ "performance profits by default. Try exploring option `spark.gluten.ras.costModel` " +
+ "for advanced usage.")
.booleanConf
.createWithDefault(false)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org