You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "NEUpanning (via GitHub)" <gi...@apache.org> on 2025/09/22 09:15:25 UTC
[PR] [GLUTEN-10773][VL] Support expression.max_compiled_regexes configuration [incubator-gluten]
NEUpanning opened a new pull request, #10776:
URL: https://github.com/apache/incubator-gluten/pull/10776
## What changes are proposed in this pull request?
Fixes https://github.com/apache/incubator-gluten/issues/10773
## How was this patch tested?
Integrated test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10773][VL] Support expression.max_compiled_regexes configuration [incubator-gluten]
Posted by "NEUpanning (via GitHub)" <gi...@apache.org>.
NEUpanning commented on PR #10776:
URL: https://github.com/apache/incubator-gluten/pull/10776#issuecomment-3383833780
@PHILO-HE Could you merge this PR? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10773][VL] Support expression.max_compiled_regexes configuration [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10776:
URL: https://github.com/apache/incubator-gluten/pull/10776#issuecomment-3317741762
https://github.com/apache/incubator-gluten/issues/10773
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10773][VL] Support expression.max_compiled_regexes configuration [incubator-gluten]
Posted by "NEUpanning (via GitHub)" <gi...@apache.org>.
NEUpanning commented on PR #10776:
URL: https://github.com/apache/incubator-gluten/pull/10776#issuecomment-3328149280
@marin-ma Could you help to take a look? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10773][VL] Support expression.max_compiled_regexes configuration [incubator-gluten]
Posted by "NEUpanning (via GitHub)" <gi...@apache.org>.
NEUpanning commented on PR #10776:
URL: https://github.com/apache/incubator-gluten/pull/10776#issuecomment-3345660834
@PHILO-HE Would you like to take a look? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10773][VL] Add support for Velox `expression.max_compiled_regexes` configuration [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE merged PR #10776:
URL: https://github.com/apache/incubator-gluten/pull/10776
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10773][VL] Support expression.max_compiled_regexes configuration [incubator-gluten]
Posted by "NEUpanning (via GitHub)" <gi...@apache.org>.
NEUpanning commented on code in PR #10776:
URL: https://github.com/apache/incubator-gluten/pull/10776#discussion_r2387151178
##########
docs/Configuration.md:
##########
@@ -75,21 +75,21 @@ nav_order: 15
| spark.gluten.sql.columnar.maxBatchSize | 4096 |
| spark.gluten.sql.columnar.overwriteByExpression | true | Enable or disable columnar v2 command overwrite by expression. |
| spark.gluten.sql.columnar.parquet.write.blockSize | 128MB |
-| spark.gluten.sql.columnar.partial.project | true | Break up one project node into 2 phases when some of the expressions are non offload-able. Phase one is a regular offloaded project transformer that evaluates the offload-able expressions in native, phase two preserves the output from phase one and evaluates the remaining non-offload-able expressions using vanilla Spark projections |
-| spark.gluten.sql.columnar.partial.generate | true | evaluates the non-offload-able HiveUDTF using vanilla Spark generator |
-| spark.gluten.sql.columnar.physicalJoinOptimizationLevel | 12 | Fallback to row operators if there are several continuous joins. |
-| spark.gluten.sql.columnar.physicalJoinOptimizeEnable | false | Enable or disable columnar physicalJoinOptimize. |
-| spark.gluten.sql.columnar.preferStreamingAggregate | true | Velox backend supports `StreamingAggregate`. `StreamingAggregate` uses the less memory as it does not need to hold all groups in memory, so it could avoid spill. When true and the child output ordering satisfies the grouping key then Gluten will choose `StreamingAggregate` as the native operator. |
-| spark.gluten.sql.columnar.project | true | Enable or disable columnar project. |
-| spark.gluten.sql.columnar.project.collapse | true | Combines two columnar project operators into one and perform alias substitution |
-| spark.gluten.sql.columnar.query.fallback.threshold | -1 | The threshold for whether query will fall back by counting the number of ColumnarToRow & vanilla leaf node. |
-| spark.gluten.sql.columnar.range | true | Enable or disable columnar range. |
-| spark.gluten.sql.columnar.replaceData | true | Enable or disable columnar v2 command replace data. |
-| spark.gluten.sql.columnar.scanOnly | false | When enabled, only scan and the filter after scan will be offloaded to native. |
-| spark.gluten.sql.columnar.shuffle | true | Enable or disable columnar shuffle. |
-| spark.gluten.sql.columnar.shuffle.celeborn.fallback.enabled | true | If enabled, fall back to ColumnarShuffleManager when celeborn service is unavailable.Otherwise, throw an exception. |
-| spark.gluten.sql.columnar.shuffle.celeborn.useRssSort | true | If true, use RSS sort implementation for Celeborn sort-based shuffle.If false, use Gluten's row-based sort implementation. Only valid when `spark.celeborn.client.spark.shuffle.writer` is set to `sort`. |
-| spark.gluten.sql.columnar.shuffle.codec | <undefined> | By default, the supported codecs are lz4 and zstd. When spark.gluten.sql.columnar.shuffle.codecBackend=qat,the supported codecs are gzip and zstd. |
+| spark.gluten.sql.columnar.partial.generate | true | Evaluates the non-offload-able HiveUDTF using vanilla Spark generator |
Review Comment:
It was generated by `dev/gen_all_config_docs.sh`. Next one who wants to generate doc will submit same change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10773][VL] Support expression.max_compiled_regexes configuration [incubator-gluten]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on code in PR #10776:
URL: https://github.com/apache/incubator-gluten/pull/10776#discussion_r2387126427
##########
docs/Configuration.md:
##########
@@ -75,21 +75,21 @@ nav_order: 15
| spark.gluten.sql.columnar.maxBatchSize | 4096 |
| spark.gluten.sql.columnar.overwriteByExpression | true | Enable or disable columnar v2 command overwrite by expression. |
| spark.gluten.sql.columnar.parquet.write.blockSize | 128MB |
-| spark.gluten.sql.columnar.partial.project | true | Break up one project node into 2 phases when some of the expressions are non offload-able. Phase one is a regular offloaded project transformer that evaluates the offload-able expressions in native, phase two preserves the output from phase one and evaluates the remaining non-offload-able expressions using vanilla Spark projections |
-| spark.gluten.sql.columnar.partial.generate | true | evaluates the non-offload-able HiveUDTF using vanilla Spark generator |
-| spark.gluten.sql.columnar.physicalJoinOptimizationLevel | 12 | Fallback to row operators if there are several continuous joins. |
-| spark.gluten.sql.columnar.physicalJoinOptimizeEnable | false | Enable or disable columnar physicalJoinOptimize. |
-| spark.gluten.sql.columnar.preferStreamingAggregate | true | Velox backend supports `StreamingAggregate`. `StreamingAggregate` uses the less memory as it does not need to hold all groups in memory, so it could avoid spill. When true and the child output ordering satisfies the grouping key then Gluten will choose `StreamingAggregate` as the native operator. |
-| spark.gluten.sql.columnar.project | true | Enable or disable columnar project. |
-| spark.gluten.sql.columnar.project.collapse | true | Combines two columnar project operators into one and perform alias substitution |
-| spark.gluten.sql.columnar.query.fallback.threshold | -1 | The threshold for whether query will fall back by counting the number of ColumnarToRow & vanilla leaf node. |
-| spark.gluten.sql.columnar.range | true | Enable or disable columnar range. |
-| spark.gluten.sql.columnar.replaceData | true | Enable or disable columnar v2 command replace data. |
-| spark.gluten.sql.columnar.scanOnly | false | When enabled, only scan and the filter after scan will be offloaded to native. |
-| spark.gluten.sql.columnar.shuffle | true | Enable or disable columnar shuffle. |
-| spark.gluten.sql.columnar.shuffle.celeborn.fallback.enabled | true | If enabled, fall back to ColumnarShuffleManager when celeborn service is unavailable.Otherwise, throw an exception. |
-| spark.gluten.sql.columnar.shuffle.celeborn.useRssSort | true | If true, use RSS sort implementation for Celeborn sort-based shuffle.If false, use Gluten's row-based sort implementation. Only valid when `spark.celeborn.client.spark.shuffle.writer` is set to `sort`. |
-| spark.gluten.sql.columnar.shuffle.codec | <undefined> | By default, the supported codecs are lz4 and zstd. When spark.gluten.sql.columnar.shuffle.codecBackend=qat,the supported codecs are gzip and zstd. |
+| spark.gluten.sql.columnar.partial.generate | true | Evaluates the non-offload-able HiveUDTF using vanilla Spark generator |
Review Comment:
seems no need change, can we remove it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10773][VL] Support expression.max_compiled_regexes configuration [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on code in PR #10776:
URL: https://github.com/apache/incubator-gluten/pull/10776#discussion_r2388412295
##########
backends-velox/src/main/scala/org/apache/gluten/config/VeloxConfig.scala:
##########
@@ -80,6 +80,8 @@ class VeloxConfig(conf: SQLConf) extends GlutenConfig(conf) {
getConf(ENABLE_ENHANCED_FEATURES)
def veloxPreferredBatchBytes: Long = getConf(COLUMNAR_VELOX_PREFERRED_BATCH_BYTES)
+
+ def maxCompiledRegexes: Int = getConf(VELOX_MAX_COMPILED_REGEXES)
Review Comment:
Do we need this? If this config will not be accessed by Scala code, maybe we can just remove it.
##########
backends-velox/src/main/scala/org/apache/gluten/config/VeloxConfig.scala:
##########
@@ -646,4 +648,10 @@ object VeloxConfig {
.internal()
.bytesConf(ByteUnit.BYTE)
.createWithDefaultString("10MB")
+
+ val VELOX_MAX_COMPILED_REGEXES =
+ buildConf("spark.gluten.sql.columnar.backend.velox.maxCompiledRegexes")
+ .doc("Controls maximum number of compiled regular expression patterns per batch.")
Review Comment:
Velox's comments for the corresponding looks clearer. Maybe, we can just copy them here.
https://github.com/facebookincubator/velox/blob/8875b4d08b26be2743bb5e79efe0548bebd08042/velox/core/QueryConfig.h#L91-L92
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org