You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "ArnavBalyan (via GitHub)" <gi...@apache.org> on 2025/05/10 15:16:13 UTC
[PR] [GLUTEN-8616] [VL] Add support for Existence Join for broadcast nested loop join [incubator-gluten]
ArnavBalyan opened a new pull request, #9588:
URL: https://github.com/apache/incubator-gluten/pull/9588
## What changes were proposed in this pull request?
- Existence join falls back for broadcast nested loop join.
- Added support for the same in Velox as leftSemiProject in the cross join/nested loop join.
- Enabled existence join for conversion. https://github.com/facebookincubator/velox/pull/12172
- Upstream PR is now merged.
## How was this patch tested?
- Existing + New UTs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8616] [VL] Add support for Existence Join for broadcast nested loop join [incubator-gluten]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on code in PR #9588:
URL: https://github.com/apache/incubator-gluten/pull/9588#discussion_r2095325195
##########
gluten-substrait/src/main/scala/org/apache/gluten/execution/BroadcastNestedLoopJoinExecTransformer.scala:
##########
@@ -186,12 +190,15 @@ abstract class BroadcastNestedLoopJoinExecTransformer(
}
(joinType, buildSide) match {
- case (LeftOuter, BuildLeft) | (RightOuter, BuildRight) =>
+ case (LeftOuter, BuildLeft) | (RightOuter, BuildRight) | (ExistenceJoin(_), BuildLeft) =>
ValidationResult.failed(s"$joinType join is not supported with $buildSide")
- case _ => ValidationResult.succeeded // continue
+ case _ =>
+ ValidationResult.succeeded
}
}
+ protected def backendSpecificJoinValidation(): Option[ValidationResult] = None
Review Comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8616] [VL] Add support for Existence Join for broadcast nested loop join [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9588:
URL: https://github.com/apache/incubator-gluten/pull/9588#issuecomment-2874715200
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8616] [VL] Add support for Existence Join for broadcast nested loop join [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on code in PR #9588:
URL: https://github.com/apache/incubator-gluten/pull/9588#discussion_r2094772275
##########
gluten-substrait/src/main/scala/org/apache/gluten/execution/BroadcastNestedLoopJoinExecTransformer.scala:
##########
@@ -186,12 +190,15 @@ abstract class BroadcastNestedLoopJoinExecTransformer(
}
(joinType, buildSide) match {
- case (LeftOuter, BuildLeft) | (RightOuter, BuildRight) =>
+ case (LeftOuter, BuildLeft) | (RightOuter, BuildRight) | (ExistenceJoin(_), BuildLeft) =>
ValidationResult.failed(s"$joinType join is not supported with $buildSide")
- case _ => ValidationResult.succeeded // continue
+ case _ =>
+ ValidationResult.succeeded
}
}
+ protected def backendSpecificJoinValidation(): Option[ValidationResult] = None
Review Comment:
Is this new method being used anywhere? If not, please remove it.
##########
backends-velox/src/test/scala/org/apache/spark/sql/execution/joins/GlutenExistenceJoinSuite.scala:
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.joins
+
+import org.apache.gluten.execution.{VeloxBroadcastNestedLoopJoinExecTransformer, VeloxWholeStageTransformerSuite}
+
+import org.apache.spark.sql.{DataFrame, Dataset, Row}
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.ExistenceJoin
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.test.SQLTestUtils
+import org.apache.spark.sql.types._
+
+class GlutenExistenceJoinSuite extends VeloxWholeStageTransformerSuite with SQLTestUtils {
+
+ test("test existence join with broadcast nested loop join") {
Review Comment:
Nit: remove "test" from test name.
##########
backends-velox/src/test/scala/org/apache/spark/sql/execution/joins/GlutenExistenceJoinSuite.scala:
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.joins
+
+import org.apache.gluten.execution.{VeloxBroadcastNestedLoopJoinExecTransformer, VeloxWholeStageTransformerSuite}
+
+import org.apache.spark.sql.{DataFrame, Dataset, Row}
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.ExistenceJoin
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.test.SQLTestUtils
+import org.apache.spark.sql.types._
+
+class GlutenExistenceJoinSuite extends VeloxWholeStageTransformerSuite with SQLTestUtils {
+
+ test("test existence join with broadcast nested loop join") {
+
+ spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "-1")
+ spark.conf.set("spark.sql.join.preferSortMergeJoin", "false")
+
+ val left: DataFrame = spark.createDataFrame(
+ sparkContext.parallelize(
+ Seq(
+ Row(1, "a"),
+ Row(2, "b"),
+ Row(3, "c")
+ )),
+ new StructType().add("id", IntegerType).add("val", StringType)
+ )
+
+ val right: DataFrame = spark.createDataFrame(
+ sparkContext.parallelize(
+ Seq(
+ Row(1, "x"),
+ Row(3, "y")
+ )),
+ new StructType().add("id", IntegerType).add("val", StringType)
+ )
+
+ val leftPlan = left.logicalPlan
+ val rightPlan = right.logicalPlan
+
+ val existsAttr = AttributeReference("exists", BooleanType, nullable = false)()
+
+ val joinCondition: Expression = LessThan(leftPlan.output(0), rightPlan.output(0))
+
+ val existenceJoin = Join(
+ left = leftPlan,
+ right = rightPlan,
+ joinType = ExistenceJoin(existsAttr),
+ condition = Some(joinCondition),
+ hint = JoinHint.NONE
+ )
+
+ val project = Project(
+ projectList = leftPlan.output :+ existsAttr,
+ child = existenceJoin
+ )
+
+ val df = Dataset.ofRows(spark, project)
+
+ assert(existenceJoin.joinType == ExistenceJoin(existsAttr))
+ assert(existenceJoin.condition.contains(joinCondition))
+ val expected = Seq(
+ Row(1, "a", true),
+ Row(2, "b", true),
+ Row(3, "c", false)
+ )
+ assert(df.collect() === expected)
+ val count = collect(df.queryExecution.executedPlan) {
+ case _: VeloxBroadcastNestedLoopJoinExecTransformer => true
+ }.size
+
+ assert(count == 1, s"Expected 1 VeloxBroadcastNestedLoopJoinExecTransformer, but found $count")
+ }
+ override protected val resourcePath: String = "N/A"
+ override protected val fileFormat: String = "N/A"
Review Comment:
Nit: move the above overriding val at the class's beginning part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8616] [VL] Add support for Existence Join for broadcast nested loop join [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9588:
URL: https://github.com/apache/incubator-gluten/pull/9588#issuecomment-2873233566
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8616] [VL] Add support for Existence Join for broadcast nested loop join [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE merged PR #9588:
URL: https://github.com/apache/incubator-gluten/pull/9588
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8616] [VL] Add support for Existence Join for broadcast nested loop join [incubator-gluten]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #9588:
URL: https://github.com/apache/incubator-gluten/pull/9588#issuecomment-2869359542
cc @zhztheplayer @PHILO-HE could you please take a look thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8616] [VL] Add support for Existence Join for broadcast nested loop join [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9588:
URL: https://github.com/apache/incubator-gluten/pull/9588#issuecomment-2868959630
https://github.com/apache/incubator-gluten/issues/8616
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8616] [VL] Add support for Existence Join for broadcast nested loop join [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9588:
URL: https://github.com/apache/incubator-gluten/pull/9588#issuecomment-2889844261
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8616] [VL] Add support for Existence Join for broadcast nested loop join [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9588:
URL: https://github.com/apache/incubator-gluten/pull/9588#issuecomment-2873262723
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8616] [VL] Add support for Existence Join for broadcast nested loop join [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9588:
URL: https://github.com/apache/incubator-gluten/pull/9588#issuecomment-2868959766
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8616] [VL] Add support for Existence Join for broadcast nested loop join [incubator-gluten]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #9588:
URL: https://github.com/apache/incubator-gluten/pull/9588#issuecomment-2889437807
cc @PHILO-HE @zhztheplayer gentle reminder thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8616] [VL] Add support for Existence Join for broadcast nested loop join [incubator-gluten]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on code in PR #9588:
URL: https://github.com/apache/incubator-gluten/pull/9588#discussion_r2095326359
##########
backends-velox/src/test/scala/org/apache/spark/sql/execution/joins/GlutenExistenceJoinSuite.scala:
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.joins
+
+import org.apache.gluten.execution.{VeloxBroadcastNestedLoopJoinExecTransformer, VeloxWholeStageTransformerSuite}
+
+import org.apache.spark.sql.{DataFrame, Dataset, Row}
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.ExistenceJoin
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.test.SQLTestUtils
+import org.apache.spark.sql.types._
+
+class GlutenExistenceJoinSuite extends VeloxWholeStageTransformerSuite with SQLTestUtils {
+
+ test("test existence join with broadcast nested loop join") {
Review Comment:
done thanks
##########
backends-velox/src/test/scala/org/apache/spark/sql/execution/joins/GlutenExistenceJoinSuite.scala:
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.joins
+
+import org.apache.gluten.execution.{VeloxBroadcastNestedLoopJoinExecTransformer, VeloxWholeStageTransformerSuite}
+
+import org.apache.spark.sql.{DataFrame, Dataset, Row}
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.ExistenceJoin
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.test.SQLTestUtils
+import org.apache.spark.sql.types._
+
+class GlutenExistenceJoinSuite extends VeloxWholeStageTransformerSuite with SQLTestUtils {
+
+ test("test existence join with broadcast nested loop join") {
+
+ spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "-1")
+ spark.conf.set("spark.sql.join.preferSortMergeJoin", "false")
+
+ val left: DataFrame = spark.createDataFrame(
+ sparkContext.parallelize(
+ Seq(
+ Row(1, "a"),
+ Row(2, "b"),
+ Row(3, "c")
+ )),
+ new StructType().add("id", IntegerType).add("val", StringType)
+ )
+
+ val right: DataFrame = spark.createDataFrame(
+ sparkContext.parallelize(
+ Seq(
+ Row(1, "x"),
+ Row(3, "y")
+ )),
+ new StructType().add("id", IntegerType).add("val", StringType)
+ )
+
+ val leftPlan = left.logicalPlan
+ val rightPlan = right.logicalPlan
+
+ val existsAttr = AttributeReference("exists", BooleanType, nullable = false)()
+
+ val joinCondition: Expression = LessThan(leftPlan.output(0), rightPlan.output(0))
+
+ val existenceJoin = Join(
+ left = leftPlan,
+ right = rightPlan,
+ joinType = ExistenceJoin(existsAttr),
+ condition = Some(joinCondition),
+ hint = JoinHint.NONE
+ )
+
+ val project = Project(
+ projectList = leftPlan.output :+ existsAttr,
+ child = existenceJoin
+ )
+
+ val df = Dataset.ofRows(spark, project)
+
+ assert(existenceJoin.joinType == ExistenceJoin(existsAttr))
+ assert(existenceJoin.condition.contains(joinCondition))
+ val expected = Seq(
+ Row(1, "a", true),
+ Row(2, "b", true),
+ Row(3, "c", false)
+ )
+ assert(df.collect() === expected)
+ val count = collect(df.queryExecution.executedPlan) {
+ case _: VeloxBroadcastNestedLoopJoinExecTransformer => true
+ }.size
+
+ assert(count == 1, s"Expected 1 VeloxBroadcastNestedLoopJoinExecTransformer, but found $count")
+ }
+ override protected val resourcePath: String = "N/A"
+ override protected val fileFormat: String = "N/A"
Review Comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org