You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "erenavsarogullari (via GitHub)" <gi...@apache.org> on 2025/01/13 01:52:33 UTC
[PR] Feat: Support array_intersect [datafusion-comet]
erenavsarogullari opened a new pull request, #1271:
URL: https://github.com/apache/datafusion-comet/pull/1271
## Which issue does this PR close?
Related to Epic: https://github.com/apache/datafusion-comet/issues/1042
array_intersect: `select array_intersect(array(1, 2, 3), array(2, 3, 4))` => `array(2, 3)`
## Rationale for this change
Defined under Epic: https://github.com/apache/datafusion-comet/issues/1042
## What changes are included in this PR?
`planner.rs:` Created DataFusion `array_intersect` physical expression from Spark physical expression,
`expr.proto:` `array_intersect` message has been added,
`QueryPlanSerde.scala:` `array_intersect` pattern matching case has been added,
`CometExpressionSuite.scala:` A new UT has been added for array_intersect function.
## How are these changes tested?
A new UT has been added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Feat: Support array_intersect [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #1271:
URL: https://github.com/apache/datafusion-comet/pull/1271#discussion_r1915556986
##########
spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala:
##########
@@ -2545,4 +2545,20 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper {
}
}
}
+
+ test("array_intersect") {
+ Seq(true, false).foreach { dictionaryEnabled =>
+ withTempDir { dir =>
+ val path = new Path(dir.toURI.toString, "test.parquet")
+ makeParquetFileAllTypes(path, dictionaryEnabled, 10000)
+ spark.read.parquet(path.toString).createOrReplaceTempView("t1")
+ checkSparkAnswerAndOperator(
+ sql("SELECT array_intersect(array(_2, _3, _4), array(_9, _10)) from t1"))
Review Comment:
Did you intend to add a commit that updates the unit test? I don't see any changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Feat: Support array_intersect [datafusion-comet]
Posted by "erenavsarogullari (via GitHub)" <gi...@apache.org>.
erenavsarogullari commented on PR #1271:
URL: https://github.com/apache/datafusion-comet/pull/1271#issuecomment-2591389516
Sure @andygrove. I can also work on https://github.com/apache/datafusion-comet/issues/1269 by assigning myself and cover `array_intersect` as part of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Feat: Support array_intersect [datafusion-comet]
Posted by "erenavsarogullari (via GitHub)" <gi...@apache.org>.
erenavsarogullari commented on PR #1271:
URL: https://github.com/apache/datafusion-comet/pull/1271#issuecomment-2600994861
> Thanks @erenavsarogullari. It would be great to have help with this. I will try and add some more notes to the issue with suggestions for how we can improve coverage.
Thanks for https://github.com/apache/datafusion-comet/pull/1308. We will need to apply same approach to other array functions after https://github.com/apache/datafusion-comet/pull/1308 is merged as part of https://github.com/apache/datafusion-comet/issues/1269. I think our scope is here to test all supported types and catch violations after passing `analysis` phase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Feat: Support array_intersect function [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #1271:
URL: https://github.com/apache/datafusion-comet/pull/1271#issuecomment-2605560537
> > Thanks @erenavsarogullari. It would be great to have help with this. I will try and add some more notes to the issue with suggestions for how we can improve coverage.
>
> Thanks for #1308. We will need to apply same approach to other array functions after #1308 is merged as part of #1269. I think our scope is here to test all supported types per array function and catch violations after passing `analysis` phase.
I agree. We are hoping to merge the comet-parquet-exec branch into main today or tomorrow, and once that is done I will go ahead and start merging the current array function PRs and then we can work on the testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Feat: Support array_intersect [datafusion-comet]
Posted by "erenavsarogullari (via GitHub)" <gi...@apache.org>.
erenavsarogullari commented on code in PR #1271:
URL: https://github.com/apache/datafusion-comet/pull/1271#discussion_r1914195082
##########
spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala:
##########
@@ -2545,4 +2545,20 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper {
}
}
}
+
+ test("array_intersect") {
+ Seq(true, false).foreach { dictionaryEnabled =>
+ withTempDir { dir =>
+ val path = new Path(dir.toURI.toString, "test.parquet")
+ makeParquetFileAllTypes(path, dictionaryEnabled, 10000)
+ spark.read.parquet(path.toString).createOrReplaceTempView("t1")
+ checkSparkAnswerAndOperator(
+ sql("SELECT array_intersect(array(_2, _3, _4), array(_9, _10)) from t1"))
Review Comment:
Thanks for the review. Updated unit test case. Spark and Comet Physical Plans are as follows:
Spark Physical Plan:
```
*(1) Project [array_intersect(array(cast(_2#1 as int), cast(_3#2 as int), _4#3), array(cast(_3#2 as int), _4#3)) AS array_intersect(array(_2, _3, _4), array(_3, _4))#44]
+- *(1) ColumnarToRow
+- FileScan parquet [_2#1,_3#2,_4#3] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/private/var/folders/jq/jhn012m16zzg7dc9lcgbdvjc0000gp/T/spark-97..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<_2:tinyint,_3:smallint,_4:int>
```
Comet Physical Plan:
```
*(1) CometColumnarToRow
+- CometProject [array_intersect(array(_2, _3, _4), array(_3, _4))#49], [array_intersect(array(cast(_2#1 as int), cast(_3#2 as int), _4#3), array(cast(_3#2 as int), _4#3)) AS array_intersect(array(_2, _3, _4), array(_3, _4))#49]
+- CometScan parquet [_2#1,_3#2,_4#3] Batched: true, DataFilters: [], Format: CometParquet, Location: InMemoryFileIndex(1 paths)[file:/private/var/folders/jq/jhn012m16zzg7dc9lcgbdvjc0000gp/T/spark-97..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<_2:tinyint,_3:smallint,_4:int>
```
Also, the execution result is as follows (by `test.parquet`):
<img width="535" alt="Query Result" src="https://github.com/user-attachments/assets/2520df45-78c5-47d1-8b0d-77b2780333a0" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Feat: Support array_intersect [datafusion-comet]
Posted by "erenavsarogullari (via GitHub)" <gi...@apache.org>.
erenavsarogullari commented on code in PR #1271:
URL: https://github.com/apache/datafusion-comet/pull/1271#discussion_r1915588010
##########
spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala:
##########
@@ -2545,4 +2545,20 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper {
}
}
}
+
+ test("array_intersect") {
+ Seq(true, false).foreach { dictionaryEnabled =>
+ withTempDir { dir =>
+ val path = new Path(dir.toURI.toString, "test.parquet")
+ makeParquetFileAllTypes(path, dictionaryEnabled, 10000)
+ spark.read.parquet(path.toString).createOrReplaceTempView("t1")
+ checkSparkAnswerAndOperator(
+ sql("SELECT array_intersect(array(_2, _3, _4), array(_9, _10)) from t1"))
Review Comment:
Yes, i have just pushed it. Thanks for the letting me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Feat: Support array_intersect [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #1271:
URL: https://github.com/apache/datafusion-comet/pull/1271#discussion_r1913324630
##########
spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala:
##########
@@ -2545,4 +2545,20 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper {
}
}
}
+
+ test("array_intersect") {
+ Seq(true, false).foreach { dictionaryEnabled =>
+ withTempDir { dir =>
+ val path = new Path(dir.toURI.toString, "test.parquet")
+ makeParquetFileAllTypes(path, dictionaryEnabled, 10000)
+ spark.read.parquet(path.toString).createOrReplaceTempView("t1")
+ checkSparkAnswerAndOperator(
+ sql("SELECT array_intersect(array(_2, _3, _4), array(_9, _10)) from t1"))
Review Comment:
It isn't obvious to me whether any of these arrays actually intersect. Perhaps you could add one that is guaranteed to intersect such as `array_intersect(array(_2, _3, _4), array(_3, _4))` or does Spark optimize that out?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Feat: Support array_intersect [datafusion-comet]
Posted by "erenavsarogullari (via GitHub)" <gi...@apache.org>.
erenavsarogullari commented on code in PR #1271:
URL: https://github.com/apache/datafusion-comet/pull/1271#discussion_r1914195082
##########
spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala:
##########
@@ -2545,4 +2545,20 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper {
}
}
}
+
+ test("array_intersect") {
+ Seq(true, false).foreach { dictionaryEnabled =>
+ withTempDir { dir =>
+ val path = new Path(dir.toURI.toString, "test.parquet")
+ makeParquetFileAllTypes(path, dictionaryEnabled, 10000)
+ spark.read.parquet(path.toString).createOrReplaceTempView("t1")
+ checkSparkAnswerAndOperator(
+ sql("SELECT array_intersect(array(_2, _3, _4), array(_9, _10)) from t1"))
Review Comment:
Thanks for the review. Updated unit test case. Spark and Comet Physical Plans are as follows:
Spark Physical Plan:
```
*(1) Project [array_intersect(array(cast(_2#1 as int), cast(_3#2 as int), _4#3), array(cast(_3#2 as int), _4#3)) AS array_intersect(array(_2, _3, _4), array(_3, _4))#44]
+- *(1) ColumnarToRow
+- FileScan parquet [_2#1,_3#2,_4#3] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/private/var/folders/jq/jhn012m16zzg7dc9lcgbdvjc0000gp/T/spark-97..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<_2:tinyint,_3:smallint,_4:int>
```
Comet Physical Plan:
```
*(1) CometColumnarToRow
+- CometProject [array_intersect(array(_2, _3, _4), array(_3, _4))#49], [array_intersect(array(cast(_2#1 as int), cast(_3#2 as int), _4#3), array(cast(_3#2 as int), _4#3)) AS array_intersect(array(_2, _3, _4), array(_3, _4))#49]
+- CometScan parquet [_2#1,_3#2,_4#3] Batched: true, DataFilters: [], Format: CometParquet, Location: InMemoryFileIndex(1 paths)[file:/private/var/folders/jq/jhn012m16zzg7dc9lcgbdvjc0000gp/T/spark-97..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<_2:tinyint,_3:smallint,_4:int>
```
Also, the execution result is as follows:
<img width="535" alt="Query Result" src="https://github.com/user-attachments/assets/2520df45-78c5-47d1-8b0d-77b2780333a0" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Feat: Support array_intersect function [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #1271:
URL: https://github.com/apache/datafusion-comet/pull/1271
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Feat: Support array_intersect [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #1271:
URL: https://github.com/apache/datafusion-comet/pull/1271#issuecomment-2599774201
> Sure @andygrove. I can also work on #1269 by assigning myself and cover `array_intersect` as part of it.
Thanks @erenavsarogullari. It would be great to have help with this. I will try and add some more notes to the issue with suggestions for how we can improve coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org