You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "zml1206 (via GitHub)" <gi...@apache.org> on 2025/03/10 08:03:52 UTC
[PR] [GLUTEN-8945][VL] Pull out duplicate projections in hash probe [incubator-gluten]
zml1206 opened a new pull request, #8946:
URL: https://github.com/apache/incubator-gluten/pull/8946
## What changes were proposed in this pull request?
(Fixes: \#8945)
## How was this patch tested?
UT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8945][VL] Pull out duplicate projections in hash probe [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8946:
URL: https://github.com/apache/incubator-gluten/pull/8946#issuecomment-2721348892
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8945][VL] Pull out duplicate projections in hash probe [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8946:
URL: https://github.com/apache/incubator-gluten/pull/8946#issuecomment-2709776037
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8945][VL] Pull out duplicate projections in hash probe [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8946:
URL: https://github.com/apache/incubator-gluten/pull/8946#issuecomment-2709727512
https://github.com/apache/incubator-gluten/issues/8945
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8945][VL] Pull out duplicate projections in hash probe [incubator-gluten]
Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 commented on code in PR #8946:
URL: https://github.com/apache/incubator-gluten/pull/8946#discussion_r1992572288
##########
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/PullOutDuplicateProject.scala:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.extension.columnar
+
+import org.apache.gluten.backendsapi.BackendsApiManager
+import org.apache.gluten.execution.{BroadcastHashJoinExecTransformerBase, LimitExecTransformer, ProjectExecTransformer}
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute, AttributeMap, AttributeSet, PredicateHelper}
+import org.apache.spark.sql.catalyst.optimizer.{BuildLeft, BuildRight}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution._
+
+import scala.collection.mutable.ArrayBuffer
+
+/**
+ * Velox does not allow duplicate projections in hash probe, this rule pull out duplicate
+ * projections.
+ */
+object PullOutDuplicateProject extends Rule[SparkPlan] with PredicateHelper {
+ override def apply(plan: SparkPlan): SparkPlan = plan.transformUp {
+ case bhj: BroadcastHashJoinExecTransformerBase =>
+ val pullOutAliases = new ArrayBuffer[Alias]()
+ val streamedPlan = rewriteStreamPlan(bhj.streamedPlan, bhj.references, pullOutAliases)
+ if (pullOutAliases.isEmpty) {
+ bhj
+ } else {
+ val aliasMap = AttributeMap(pullOutAliases.map(a => a.toAttribute -> a))
+ val newProjectList = bhj.output.map(attr => aliasMap.getOrElse(attr, attr))
+ val (newLeft, newRight) = bhj.joinBuildSide match {
+ case BuildLeft => (bhj.left, streamedPlan)
+ case BuildRight => (streamedPlan, bhj.right)
+ }
+
+ val newBhj =
+ BackendsApiManager.getSparkPlanExecApiInstance.genBroadcastHashJoinExecTransformer(
+ bhj.leftKeys,
+ bhj.rightKeys,
+ bhj.hashJoinType,
+ bhj.joinBuildSide,
+ bhj.condition,
+ newLeft,
+ newRight,
+ bhj.genJoinParametersInternal()._2 == 1)
+ ProjectExecTransformer(newProjectList, newBhj)
+ }
+ }
+
+ def rewriteStreamPlan(
Review Comment:
Added.
##########
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/PullOutDuplicateProject.scala:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.extension.columnar
+
+import org.apache.gluten.backendsapi.BackendsApiManager
+import org.apache.gluten.execution.{BroadcastHashJoinExecTransformerBase, LimitExecTransformer, ProjectExecTransformer}
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute, AttributeMap, AttributeSet, PredicateHelper}
+import org.apache.spark.sql.catalyst.optimizer.{BuildLeft, BuildRight}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution._
+
+import scala.collection.mutable.ArrayBuffer
+
+/**
+ * Velox does not allow duplicate projections in hash probe, this rule pull out duplicate
+ * projections.
+ */
Review Comment:
Added in pr description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8945][VL] Pull out duplicate projections for HashProbe and FilterProject [incubator-gluten]
Posted by "rui-mo (via GitHub)" <gi...@apache.org>.
rui-mo merged PR #8946:
URL: https://github.com/apache/incubator-gluten/pull/8946
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8945][VL] Pull out duplicate projections for HashProbe and FilterProject [incubator-gluten]
Posted by "rui-mo (via GitHub)" <gi...@apache.org>.
rui-mo commented on code in PR #8946:
URL: https://github.com/apache/incubator-gluten/pull/8946#discussion_r2000705319
##########
backends-velox/src/main/scala/org/apache/gluten/extension/PullOutDuplicateProject.scala:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.extension
+
+import org.apache.gluten.execution.{BroadcastHashJoinExecTransformer, FilterExecTransformer, LimitExecTransformer, ProjectExecTransformer}
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute, AttributeMap, AttributeSet, PredicateHelper}
+import org.apache.spark.sql.catalyst.optimizer.{BuildLeft, BuildRight}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution._
+
+import scala.collection.mutable.ArrayBuffer
+
+/**
+ * Velox does not allow duplicate projections in HashProbe and FilterProject, this rule pull out
Review Comment:
> Velox does not allow duplicate projections in HashProbe and FilterProject
Could you provide a reference in Velox for this description, and is there any other operation having similar limitation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8945][VL] Pull out duplicate projections for HashProbe and FilterProject [incubator-gluten]
Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 commented on code in PR #8946:
URL: https://github.com/apache/incubator-gluten/pull/8946#discussion_r2000717698
##########
backends-velox/src/main/scala/org/apache/gluten/extension/PullOutDuplicateProject.scala:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.extension
+
+import org.apache.gluten.execution.{BroadcastHashJoinExecTransformer, FilterExecTransformer, LimitExecTransformer, ProjectExecTransformer}
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute, AttributeMap, AttributeSet, PredicateHelper}
+import org.apache.spark.sql.catalyst.optimizer.{BuildLeft, BuildRight}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution._
+
+import scala.collection.mutable.ArrayBuffer
+
+/**
+ * Velox does not allow duplicate projections in HashProbe and FilterProject, this rule pull out
Review Comment:
https://github.com/facebookincubator/velox/issues/12484#issuecomment-2704265309,
I have only found these two so far, I am not sure about the others.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8945][VL] Pull out duplicate projections in hash probe [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8946:
URL: https://github.com/apache/incubator-gluten/pull/8946#issuecomment-2719637171
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8945][VL] Pull out duplicate projections in hash probe [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8946:
URL: https://github.com/apache/incubator-gluten/pull/8946#issuecomment-2709728098
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8945][VL] Pull out duplicate projections in hash probe [incubator-gluten]
Posted by "rui-mo (via GitHub)" <gi...@apache.org>.
rui-mo commented on code in PR #8946:
URL: https://github.com/apache/incubator-gluten/pull/8946#discussion_r1991287984
##########
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/PullOutDuplicateProject.scala:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.extension.columnar
+
+import org.apache.gluten.backendsapi.BackendsApiManager
+import org.apache.gluten.execution.{BroadcastHashJoinExecTransformerBase, LimitExecTransformer, ProjectExecTransformer}
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute, AttributeMap, AttributeSet, PredicateHelper}
+import org.apache.spark.sql.catalyst.optimizer.{BuildLeft, BuildRight}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution._
+
+import scala.collection.mutable.ArrayBuffer
+
+/**
+ * Velox does not allow duplicate projections in hash probe, this rule pull out duplicate
+ * projections.
+ */
+object PullOutDuplicateProject extends Rule[SparkPlan] with PredicateHelper {
+ override def apply(plan: SparkPlan): SparkPlan = plan.transformUp {
+ case bhj: BroadcastHashJoinExecTransformerBase =>
+ val pullOutAliases = new ArrayBuffer[Alias]()
+ val streamedPlan = rewriteStreamPlan(bhj.streamedPlan, bhj.references, pullOutAliases)
+ if (pullOutAliases.isEmpty) {
+ bhj
+ } else {
+ val aliasMap = AttributeMap(pullOutAliases.map(a => a.toAttribute -> a))
+ val newProjectList = bhj.output.map(attr => aliasMap.getOrElse(attr, attr))
+ val (newLeft, newRight) = bhj.joinBuildSide match {
+ case BuildLeft => (bhj.left, streamedPlan)
+ case BuildRight => (streamedPlan, bhj.right)
+ }
+
+ val newBhj =
+ BackendsApiManager.getSparkPlanExecApiInstance.genBroadcastHashJoinExecTransformer(
+ bhj.leftKeys,
+ bhj.rightKeys,
+ bhj.hashJoinType,
+ bhj.joinBuildSide,
+ bhj.condition,
+ newLeft,
+ newRight,
+ bhj.genJoinParametersInternal()._2 == 1)
+ ProjectExecTransformer(newProjectList, newBhj)
+ }
+ }
+
+ def rewriteStreamPlan(
Review Comment:
Please add comments for how stream plan is rewritten.
##########
gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/PullOutDuplicateProject.scala:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.extension.columnar
+
+import org.apache.gluten.backendsapi.BackendsApiManager
+import org.apache.gluten.execution.{BroadcastHashJoinExecTransformerBase, LimitExecTransformer, ProjectExecTransformer}
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute, AttributeMap, AttributeSet, PredicateHelper}
+import org.apache.spark.sql.catalyst.optimizer.{BuildLeft, BuildRight}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution._
+
+import scala.collection.mutable.ArrayBuffer
+
+/**
+ * Velox does not allow duplicate projections in hash probe, this rule pull out duplicate
+ * projections.
+ */
Review Comment:
Perhaps elaborate on what was the plan like, and what is the new plan like.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org