You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "zml1206 (via GitHub)" <gi...@apache.org> on 2024/12/04 09:46:32 UTC
[PR] [CORE] Simplify code of offload scan [incubator-gluten]
zml1206 opened a new pull request, #8144:
URL: https://github.com/apache/incubator-gluten/pull/8144
## What changes were proposed in this pull request?
## How was this patch tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Simplify code of offload scan [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8144:
URL: https://github.com/apache/incubator-gluten/pull/8144#issuecomment-2516721713
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Simplify code of offload scan [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8144:
URL: https://github.com/apache/incubator-gluten/pull/8144#issuecomment-2518982762
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Simplify code of offload scan [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8144:
URL: https://github.com/apache/incubator-gluten/pull/8144#issuecomment-2519340867
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Simplify code of offload scan [incubator-gluten]
Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 commented on code in PR #8144:
URL: https://github.com/apache/incubator-gluten/pull/8144#discussion_r1870739172
##########
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/enumerated/RasOffload.scala:
##########
@@ -118,21 +119,29 @@ object RasOffload {
// 4. Validate current node. If passed, offload it.
validator.validate(from) match {
case Validator.Passed =>
- val offloadedPlan = base.offload(from)
- val offloadedNodes = offloadedPlan.collect[ValidatablePlan] {
- case t: ValidatablePlan => t
- }
- val outComes = offloadedNodes.map(_.doValidate()).filter(!_.ok())
- if (outComes.nonEmpty) {
- // 5. If native validation fails on at least one of the offloaded nodes, return
- // the original one.
- //
- // TODO: Tag the original plan with fallback reason. This is a non-trivial work
- // in RAS as the query plan we got here may be a copy so may not propagate tags
- // to original plan.
- from
- } else {
- offloadedPlan
+ try {
+ val offloadedPlan = base.offload(from)
+ val offloadedNodes = offloadedPlan.collect[ValidatablePlan] {
+ case t: ValidatablePlan => t
+ }
+ val outComes = offloadedNodes.map(_.doValidate()).filter(!_.ok())
+ if (outComes.nonEmpty) {
+ // 5. If native validation fails on at least one of the offloaded nodes, return
+ // the original one.
+ //
+ // TODO: Tag the original plan with fallback reason. This is a non-trivial work
+ // in RAS as the query plan we got here may be a copy so may not propagate tags
+ // to original plan.
+ from
+ } else {
+ offloadedPlan
+ }
+ } catch {
+ case _: GlutenNotSupportException =>
Review Comment:
Keep and disable RAS consistent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Simplify code of offload scan [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8144:
URL: https://github.com/apache/incubator-gluten/pull/8144#issuecomment-2519284840
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Simplify code of offload scan [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8144:
URL: https://github.com/apache/incubator-gluten/pull/8144#issuecomment-2519968297
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Simplify code of offload scan [incubator-gluten]
Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 commented on code in PR #8144:
URL: https://github.com/apache/incubator-gluten/pull/8144#discussion_r1870844498
##########
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/enumerated/RasOffload.scala:
##########
@@ -118,21 +119,29 @@ object RasOffload {
// 4. Validate current node. If passed, offload it.
validator.validate(from) match {
case Validator.Passed =>
- val offloadedPlan = base.offload(from)
- val offloadedNodes = offloadedPlan.collect[ValidatablePlan] {
- case t: ValidatablePlan => t
- }
- val outComes = offloadedNodes.map(_.doValidate()).filter(!_.ok())
- if (outComes.nonEmpty) {
- // 5. If native validation fails on at least one of the offloaded nodes, return
- // the original one.
- //
- // TODO: Tag the original plan with fallback reason. This is a non-trivial work
- // in RAS as the query plan we got here may be a copy so may not propagate tags
- // to original plan.
- from
- } else {
- offloadedPlan
+ try {
+ val offloadedPlan = base.offload(from)
+ val offloadedNodes = offloadedPlan.collect[ValidatablePlan] {
+ case t: ValidatablePlan => t
+ }
+ val outComes = offloadedNodes.map(_.doValidate()).filter(!_.ok())
+ if (outComes.nonEmpty) {
+ // 5. If native validation fails on at least one of the offloaded nodes, return
+ // the original one.
+ //
+ // TODO: Tag the original plan with fallback reason. This is a non-trivial work
+ // in RAS as the query plan we got here may be a copy so may not propagate tags
+ // to original plan.
+ from
+ } else {
+ offloadedPlan
+ }
+ } catch {
+ case _: GlutenNotSupportException =>
Review Comment:
I didn't find the place where GlutenNotSupportException is thrown in the offload rule. `doValidate()` contains `doValidateInternal()` and `doSchemaValidate()`, `doValidateInternal()` has been caught, `doSchemaValidate()` will not trigger an exception, so restore it？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Simplify code of offload scan [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8144:
URL: https://github.com/apache/incubator-gluten/pull/8144#issuecomment-2519286803
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Simplify code of offload scan [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8144:
URL: https://github.com/apache/incubator-gluten/pull/8144#issuecomment-2516721176
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICEGJ-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
Could you open an issue for this pull request on Github Issues?
https://github.com/apache/incubator-gluten/issues
Then could you also rename ***commit message*** and ***pull request title*** in the following format?
[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}
See also:
* [Other pull requests](https://github.com/apache/incubator-gluten/pulls/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Simplify code of offload scan [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #8144:
URL: https://github.com/apache/incubator-gluten/pull/8144#discussion_r1870775660
##########
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/enumerated/RasOffload.scala:
##########
@@ -118,21 +119,29 @@ object RasOffload {
// 4. Validate current node. If passed, offload it.
validator.validate(from) match {
case Validator.Passed =>
- val offloadedPlan = base.offload(from)
- val offloadedNodes = offloadedPlan.collect[ValidatablePlan] {
- case t: ValidatablePlan => t
- }
- val outComes = offloadedNodes.map(_.doValidate()).filter(!_.ok())
- if (outComes.nonEmpty) {
- // 5. If native validation fails on at least one of the offloaded nodes, return
- // the original one.
- //
- // TODO: Tag the original plan with fallback reason. This is a non-trivial work
- // in RAS as the query plan we got here may be a copy so may not propagate tags
- // to original plan.
- from
- } else {
- offloadedPlan
+ try {
+ val offloadedPlan = base.offload(from)
+ val offloadedNodes = offloadedPlan.collect[ValidatablePlan] {
+ case t: ValidatablePlan => t
+ }
+ val outComes = offloadedNodes.map(_.doValidate()).filter(!_.ok())
+ if (outComes.nonEmpty) {
+ // 5. If native validation fails on at least one of the offloaded nodes, return
+ // the original one.
+ //
+ // TODO: Tag the original plan with fallback reason. This is a non-trivial work
+ // in RAS as the query plan we got here may be a copy so may not propagate tags
+ // to original plan.
+ from
+ } else {
+ offloadedPlan
+ }
+ } catch {
+ case _: GlutenNotSupportException =>
Review Comment:
Would you help check the corresponding logic in legacy planner to see whether we should only catch exceptions from `doValidate()` or catch more code like this way? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Simplify code of offload scan [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #8144:
URL: https://github.com/apache/incubator-gluten/pull/8144#discussion_r1871450286
##########
gluten-substrait/src/main/scala/org/apache/gluten/backendsapi/BackendSettingsApi.scala:
##########
@@ -121,6 +121,8 @@ trait BackendSettingsApi {
def supportCartesianProductExec(): Boolean = false
+ def supportCartesianProductExecWithCondition(): Boolean = true
Review Comment:
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Simplify code of offload scan [incubator-gluten]
Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 merged PR #8144:
URL: https://github.com/apache/incubator-gluten/pull/8144
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Simplify code of offload scan [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8144:
URL: https://github.com/apache/incubator-gluten/pull/8144#issuecomment-2519605139
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Simplify code of offload scan [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #8144:
URL: https://github.com/apache/incubator-gluten/pull/8144#discussion_r1870856378
##########
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/enumerated/RasOffload.scala:
##########
@@ -118,21 +119,29 @@ object RasOffload {
// 4. Validate current node. If passed, offload it.
validator.validate(from) match {
case Validator.Passed =>
- val offloadedPlan = base.offload(from)
- val offloadedNodes = offloadedPlan.collect[ValidatablePlan] {
- case t: ValidatablePlan => t
- }
- val outComes = offloadedNodes.map(_.doValidate()).filter(!_.ok())
- if (outComes.nonEmpty) {
- // 5. If native validation fails on at least one of the offloaded nodes, return
- // the original one.
- //
- // TODO: Tag the original plan with fallback reason. This is a non-trivial work
- // in RAS as the query plan we got here may be a copy so may not propagate tags
- // to original plan.
- from
- } else {
- offloadedPlan
+ try {
+ val offloadedPlan = base.offload(from)
+ val offloadedNodes = offloadedPlan.collect[ValidatablePlan] {
+ case t: ValidatablePlan => t
+ }
+ val outComes = offloadedNodes.map(_.doValidate()).filter(!_.ok())
+ if (outComes.nonEmpty) {
+ // 5. If native validation fails on at least one of the offloaded nodes, return
+ // the original one.
+ //
+ // TODO: Tag the original plan with fallback reason. This is a non-trivial work
+ // in RAS as the query plan we got here may be a copy so may not propagate tags
+ // to original plan.
+ from
+ } else {
+ offloadedPlan
+ }
+ } catch {
+ case _: GlutenNotSupportException =>
Review Comment:
I see. So in this PR we can shrink the range of the catch block to only catch `doValidate()` to match legacy planner. Perhaps in future we should make all exceptions be caught in `doValidate` internally to make `doValidate` exception-free.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org