You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "Zouxxyy (via GitHub)" <gi...@apache.org> on 2025/11/13 07:37:14 UTC
[PR] [CORE] Remove pushed down filter in FilterExecTransformer [incubator-gluten]
Zouxxyy opened a new pull request, #11076:
URL: https://github.com/apache/incubator-gluten/pull/11076
<!--
Thank you for submitting a pull request! Here are some tips:
1. For first-time contributors, please read our contributing guide:
https://github.com/apache/incubator-gluten/blob/main/CONTRIBUTING.md
2. If necessary, create a GitHub issue for discussion beforehand to avoid duplicate work.
3. If the PR is specific to a single backend, include [VL] or [CH] in the PR title to indicate the
Velox or ClickHouse backend, respectively.
4. If the PR is not ready for review, please mark it as a draft.
-->
## What changes are proposed in this pull request?
Add a new rule `RemovePushedDownFilter` to FilterExecTransformer
before
<img width="350" height="187" alt="image" src="https://github.com/user-attachments/assets/b6372e79-fe7a-4e4a-812a-56d5c962a60b" />
after
<img width="244" height="144" alt="image" src="https://github.com/user-attachments/assets/f33aa552-fb75-4f0a-91a6-46f00e97bbb1" />
And the pushed native filter can be seen in the scan node
<img width="295" height="180" alt="image" src="https://github.com/user-attachments/assets/9f00f771-adf6-458b-919d-7c7118b66899" />
## How was this patch tested?
<!--
Describe how the changes were tested, if applicable.
Include new tests to validate the functionality, if necessary.
For UI-related changes, attach screenshots to demonstrate the updates.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Remove pushed down filter in FilterExecTransformer [incubator-gluten]
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on code in PR #11076:
URL: https://github.com/apache/incubator-gluten/pull/11076#discussion_r2522098619
##########
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/RemovePushedDownFilter.scala:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.extension.columnar
+
+import org.apache.gluten.backendsapi.BackendsApiManager
+import org.apache.gluten.execution.{BasicScanExecTransformer, FilterExecTransformerBase, FilterHandler}
+
+import org.apache.spark.sql.catalyst.expressions.{And, Expression, PredicateHelper}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.SparkPlan
+
+/** Remove the filter that have been pushed down to native scan. */
+object RemovePushedDownFilter extends Rule[SparkPlan] with PredicateHelper {
+
+ override def apply(plan: SparkPlan): SparkPlan = plan.transform {
+ case filter: FilterExecTransformerBase if filter.cond != null =>
+ val scanFilters = filter.child match {
+ // Get the filters including the manually pushed down ones.
+ case basicScanExecTransformer: BasicScanExecTransformer =>
+ basicScanExecTransformer.filterExprs()
+ // For fallback scan, we need to keep original filter.
+ case _ =>
+ Seq.empty[Expression]
+ }
+ if (scanFilters.isEmpty) {
+ filter
+ } else {
+ val remainingFilters =
+ FilterHandler.getRemainingFilters(scanFilters, splitConjunctivePredicates(filter.cond))
+ val newCond = remainingFilters.reduceLeftOption(And).orNull
Review Comment:
Maybe we can remove the filter node, once the newCond is null
##########
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/RemovePushedDownFilter.scala:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.extension.columnar
+
+import org.apache.gluten.backendsapi.BackendsApiManager
+import org.apache.gluten.execution.{BasicScanExecTransformer, FilterExecTransformerBase, FilterHandler}
+
+import org.apache.spark.sql.catalyst.expressions.{And, Expression, PredicateHelper}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.SparkPlan
+
+/** Remove the filter that have been pushed down to native scan. */
+object RemovePushedDownFilter extends Rule[SparkPlan] with PredicateHelper {
+
+ override def apply(plan: SparkPlan): SparkPlan = plan.transform {
+ case filter: FilterExecTransformerBase if filter.cond != null =>
+ val scanFilters = filter.child match {
+ // Get the filters including the manually pushed down ones.
+ case basicScanExecTransformer: BasicScanExecTransformer =>
+ basicScanExecTransformer.filterExprs()
+ // For fallback scan, we need to keep original filter.
+ case _ =>
+ Seq.empty[Expression]
+ }
+ if (scanFilters.isEmpty) {
+ filter
+ } else {
+ val remainingFilters =
+ FilterHandler.getRemainingFilters(scanFilters, splitConjunctivePredicates(filter.cond))
+ val newCond = remainingFilters.reduceLeftOption(And).orNull
Review Comment:
Maybe we can remove the filter node, once the newCond is null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Remove pushed down filter in FilterExecTransformer [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11076:
URL: https://github.com/apache/incubator-gluten/pull/11076#issuecomment-3536323993
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Remove pushed down filter in FilterExecTransformer [incubator-gluten]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on PR #11076:
URL: https://github.com/apache/incubator-gluten/pull/11076#issuecomment-3530395844
Do `pushedDownFilters` always work for all datasource scans? I'm worried that some datasource scan implementations might ignore some unsupported pushedDownFilters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Remove pushed down filter in FilterExecTransformer [incubator-gluten]
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on code in PR #11076:
URL: https://github.com/apache/incubator-gluten/pull/11076#discussion_r2522107399
##########
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/RemovePushedDownFilter.scala:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.extension.columnar
+
+import org.apache.gluten.backendsapi.BackendsApiManager
+import org.apache.gluten.execution.{BasicScanExecTransformer, FilterExecTransformerBase, FilterHandler}
+
+import org.apache.spark.sql.catalyst.expressions.{And, Expression, PredicateHelper}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.SparkPlan
+
+/** Remove the filter that have been pushed down to native scan. */
+object RemovePushedDownFilter extends Rule[SparkPlan] with PredicateHelper {
+
+ override def apply(plan: SparkPlan): SparkPlan = plan.transform {
+ case filter: FilterExecTransformerBase if filter.cond != null =>
+ val scanFilters = filter.child match {
+ // Get the filters including the manually pushed down ones.
+ case basicScanExecTransformer: BasicScanExecTransformer =>
+ basicScanExecTransformer.filterExprs()
+ // For fallback scan, we need to keep original filter.
+ case _ =>
+ Seq.empty[Expression]
+ }
+ if (scanFilters.isEmpty) {
+ filter
+ } else {
+ val remainingFilters =
+ FilterHandler.getRemainingFilters(scanFilters, splitConjunctivePredicates(filter.cond))
+ val newCond = remainingFilters.reduceLeftOption(And).orNull
Review Comment:
Perhaps we could remove the filterNode if newCond is null. If guys agree, I can include this change in this PR CC @zhztheplayer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Remove pushed down filter in FilterExecTransformer [incubator-gluten]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on PR #11076:
URL: https://github.com/apache/incubator-gluten/pull/11076#issuecomment-3530544176
> > Do `pushedDownFilters` always work for all datasource scans? I'm worried that some datasource scan implementations might ignore some unsupported pushedDownFilters.
>
> The logic should be the same as before, because we were already using getRemainingCondition to generate the Filter Transform previously. I just made the execution plan more accurately reflect its actual behavior. For example, users might notice there's a filter node, but in reality, it doesn't do anything.
Got it, thank you for your explanation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Remove pushed down filter in FilterExecTransformer [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #11076:
URL: https://github.com/apache/incubator-gluten/pull/11076#discussion_r2526910400
##########
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/RemovePushedDownFilter.scala:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.extension.columnar
+
+import org.apache.gluten.backendsapi.BackendsApiManager
+import org.apache.gluten.execution.{BasicScanExecTransformer, FilterExecTransformerBase, FilterHandler}
+
+import org.apache.spark.sql.catalyst.expressions.{And, Expression, PredicateHelper}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.SparkPlan
+
+/** Remove the filter that have been pushed down to native scan. */
+object RemovePushedDownFilter extends Rule[SparkPlan] with PredicateHelper {
+
+ override def apply(plan: SparkPlan): SparkPlan = plan.transform {
+ case filter: FilterExecTransformerBase if filter.cond != null =>
+ val scanFilters = filter.child match {
+ // Get the filters including the manually pushed down ones.
+ case basicScanExecTransformer: BasicScanExecTransformer =>
+ basicScanExecTransformer.filterExprs()
+ // For fallback scan, we need to keep original filter.
+ case _ =>
+ Seq.empty[Expression]
+ }
+ if (scanFilters.isEmpty) {
+ filter
+ } else {
+ val remainingFilters =
+ FilterHandler.getRemainingFilters(scanFilters, splitConjunctivePredicates(filter.cond))
+ val newCond = remainingFilters.reduceLeftOption(And).orNull
Review Comment:
> Perhaps we could remove the filterNode if newCond is null.
I previously attempted something similar, but ended up facing some attribute mismatch errors from Catalyst.
I didn't look into that issue too much, so feel free to try from your end if you're interested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Remove pushed down filter in FilterExecTransformer [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11076:
URL: https://github.com/apache/incubator-gluten/pull/11076#issuecomment-3536328220
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Remove pushed down filter in FilterExecTransformer [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11076:
URL: https://github.com/apache/incubator-gluten/pull/11076#issuecomment-3536190014
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Remove pushed down filter in FilterExecTransformer [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11076:
URL: https://github.com/apache/incubator-gluten/pull/11076#issuecomment-3526086320
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Remove pushed down filter in FilterExecTransformer [incubator-gluten]
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on code in PR #11076:
URL: https://github.com/apache/incubator-gluten/pull/11076#discussion_r2522106135
##########
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/RemovePushedDownFilter.scala:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.extension.columnar
+
+import org.apache.gluten.backendsapi.BackendsApiManager
+import org.apache.gluten.execution.{BasicScanExecTransformer, FilterExecTransformerBase, FilterHandler}
+
+import org.apache.spark.sql.catalyst.expressions.{And, Expression, PredicateHelper}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.SparkPlan
+
+/** Remove the filter that have been pushed down to native scan. */
+object RemovePushedDownFilter extends Rule[SparkPlan] with PredicateHelper {
+
+ override def apply(plan: SparkPlan): SparkPlan = plan.transform {
+ case filter: FilterExecTransformerBase if filter.cond != null =>
+ val scanFilters = filter.child match {
+ // Get the filters including the manually pushed down ones.
+ case basicScanExecTransformer: BasicScanExecTransformer =>
+ basicScanExecTransformer.filterExprs()
+ // For fallback scan, we need to keep original filter.
+ case _ =>
+ Seq.empty[Expression]
+ }
+ if (scanFilters.isEmpty) {
+ filter
+ } else {
+ val remainingFilters =
+ FilterHandler.getRemainingFilters(scanFilters, splitConjunctivePredicates(filter.cond))
+ val newCond = remainingFilters.reduceLeftOption(And).orNull
+ BackendsApiManager.getSparkPlanExecApiInstance
Review Comment:
Perhaps we could remove the filterNode if newCond is null. If guys agree, I can include this change in this PR
##########
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/RemovePushedDownFilter.scala:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.extension.columnar
+
+import org.apache.gluten.backendsapi.BackendsApiManager
+import org.apache.gluten.execution.{BasicScanExecTransformer, FilterExecTransformerBase, FilterHandler}
+
+import org.apache.spark.sql.catalyst.expressions.{And, Expression, PredicateHelper}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.SparkPlan
+
+/** Remove the filter that have been pushed down to native scan. */
+object RemovePushedDownFilter extends Rule[SparkPlan] with PredicateHelper {
+
+ override def apply(plan: SparkPlan): SparkPlan = plan.transform {
+ case filter: FilterExecTransformerBase if filter.cond != null =>
+ val scanFilters = filter.child match {
+ // Get the filters including the manually pushed down ones.
+ case basicScanExecTransformer: BasicScanExecTransformer =>
+ basicScanExecTransformer.filterExprs()
+ // For fallback scan, we need to keep original filter.
+ case _ =>
+ Seq.empty[Expression]
+ }
+ if (scanFilters.isEmpty) {
+ filter
+ } else {
+ val remainingFilters =
+ FilterHandler.getRemainingFilters(scanFilters, splitConjunctivePredicates(filter.cond))
+ val newCond = remainingFilters.reduceLeftOption(And).orNull
+ BackendsApiManager.getSparkPlanExecApiInstance
Review Comment:
Perhaps we could remove the filterNode if newCond is null. If guys agree, I can include this change in this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Remove pushed down filter in FilterExecTransformer [incubator-gluten]
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on PR #11076:
URL: https://github.com/apache/incubator-gluten/pull/11076#issuecomment-3530460100
> Do `pushedDownFilters` always work for all datasource scans? I'm worried that some datasource scan implementations might ignore some unsupported pushedDownFilters.
The logic should be the same as before, because we were already using getRemainingCondition to generate the Filter Transform previously. I just made the execution plan more accurately reflect its actual behavior. For example, users might notice there's a filter node, but in reality, it doesn't do anything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Remove pushed down filter in FilterExecTransformer [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11076:
URL: https://github.com/apache/incubator-gluten/pull/11076#issuecomment-3701259109
This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Remove pushed down filter in FilterExecTransformer [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11076:
URL: https://github.com/apache/incubator-gluten/pull/11076#issuecomment-3526108422
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Remove pushed down filter in FilterExecTransformer [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11076:
URL: https://github.com/apache/incubator-gluten/pull/11076#issuecomment-3536301615
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Remove pushed down filter in FilterExecTransformer [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11076:
URL: https://github.com/apache/incubator-gluten/pull/11076#issuecomment-3536174732
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Remove pushed down filter in FilterExecTransformer [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #11076: [CORE] Remove pushed down filter in FilterExecTransformer
URL: https://github.com/apache/incubator-gluten/pull/11076
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Remove pushed down filter in FilterExecTransformer [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11076:
URL: https://github.com/apache/incubator-gluten/pull/11076#issuecomment-3733830722
This PR was auto-closed because it has been stalled for 10 days with no activity. Please feel free to reopen if it is still valid. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Remove pushed down filter in FilterExecTransformer [incubator-gluten]
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on code in PR #11076:
URL: https://github.com/apache/incubator-gluten/pull/11076#discussion_r2529689419
##########
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/RemovePushedDownFilter.scala:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.extension.columnar
+
+import org.apache.gluten.backendsapi.BackendsApiManager
+import org.apache.gluten.execution.{BasicScanExecTransformer, FilterExecTransformerBase, FilterHandler}
+
+import org.apache.spark.sql.catalyst.expressions.{And, Expression, PredicateHelper}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.SparkPlan
+
+/** Remove the filter that have been pushed down to native scan. */
+object RemovePushedDownFilter extends Rule[SparkPlan] with PredicateHelper {
+
+ override def apply(plan: SparkPlan): SparkPlan = plan.transform {
+ case filter: FilterExecTransformerBase if filter.cond != null =>
+ val scanFilters = filter.child match {
+ // Get the filters including the manually pushed down ones.
+ case basicScanExecTransformer: BasicScanExecTransformer =>
+ basicScanExecTransformer.filterExprs()
+ // For fallback scan, we need to keep original filter.
+ case _ =>
+ Seq.empty[Expression]
+ }
+ if (scanFilters.isEmpty) {
+ filter
+ } else {
+ val remainingFilters =
+ FilterHandler.getRemainingFilters(scanFilters, splitConjunctivePredicates(filter.cond))
+ val newCond = remainingFilters.reduceLeftOption(And).orNull
Review Comment:
> facing some attribute mismatch errors from Catalyst.
@zhztheplayer I checked the logic of FilterExec and confirmed that its output indeed changes—it equals the child's output plus some NOT NULL attributes added by the condition. Therefore, I made the following modifications:
- Extracted a helper method `buildNewOutput(output, conds)` from FilterExec.
- If pushedFilter nonEmpty:
- Set ScanNode's newOutput = buildNewOutput(output, pushedFilter).
- Set FilterNode's newCond = cond -- pushedFilter. (If newCond == null, remove the filter node.)
I believe this better reflects Node's correct behavior, although perhaps all golden files should be updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Remove pushed down filter in FilterExecTransformer [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11076:
URL: https://github.com/apache/incubator-gluten/pull/11076#issuecomment-3526143040
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Remove pushed down filter in FilterExecTransformer [incubator-gluten]
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on code in PR #11076:
URL: https://github.com/apache/incubator-gluten/pull/11076#discussion_r2522107399
##########
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/RemovePushedDownFilter.scala:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.extension.columnar
+
+import org.apache.gluten.backendsapi.BackendsApiManager
+import org.apache.gluten.execution.{BasicScanExecTransformer, FilterExecTransformerBase, FilterHandler}
+
+import org.apache.spark.sql.catalyst.expressions.{And, Expression, PredicateHelper}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.SparkPlan
+
+/** Remove the filter that have been pushed down to native scan. */
+object RemovePushedDownFilter extends Rule[SparkPlan] with PredicateHelper {
+
+ override def apply(plan: SparkPlan): SparkPlan = plan.transform {
+ case filter: FilterExecTransformerBase if filter.cond != null =>
+ val scanFilters = filter.child match {
+ // Get the filters including the manually pushed down ones.
+ case basicScanExecTransformer: BasicScanExecTransformer =>
+ basicScanExecTransformer.filterExprs()
+ // For fallback scan, we need to keep original filter.
+ case _ =>
+ Seq.empty[Expression]
+ }
+ if (scanFilters.isEmpty) {
+ filter
+ } else {
+ val remainingFilters =
+ FilterHandler.getRemainingFilters(scanFilters, splitConjunctivePredicates(filter.cond))
+ val newCond = remainingFilters.reduceLeftOption(And).orNull
Review Comment:
Perhaps we could remove the filterNode if newCond is null. If guys agree, I can include this change in this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org