You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "comphead (via GitHub)" <gi...@apache.org> on 2025/06/24 14:49:16 UTC
[I] `array_contains` falls back to Spark in case of empty Array [datafusion-comet]
comphead opened a new issue, #1929:
URL: https://github.com/apache/datafusion-comet/issues/1929
### Describe the bug
After reviewing https://github.com/apache/datafusion/pull/16529
I added empty array test to Comet
```
test("array_contains") {
withSQLConf(CometConf.COMET_EXPR_ALLOW_INCOMPATIBLE.key -> "true") {
withTempDir { dir =>
val path = new Path(dir.toURI.toString, "test.parquet")
makeParquetFileAllTypes(path, dictionaryEnabled = false, n = 10000)
spark.read.parquet(path.toString).createOrReplaceTempView("t1");
checkSparkAnswerAndOperator(
spark.sql("SELECT array_contains(array(_2, _3, _4), _2) FROM t1"))
checkSparkAnswerAndOperator(
spark.sql("SELECT array_contains((CASE WHEN _2 =_3 THEN array(_4) END), _4) FROM t1"));
checkSparkAnswerAndOperator(spark.sql("SELECT array_contains(cast(array() as array<int>), _2) FROM t1"))
}
}
}
```
and it fails with fallback
```
Expected only Comet native operators, but found Project.
```
### Steps to reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `array_contains` falls back to Spark in case of empty Array [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1929:
URL: https://github.com/apache/datafusion-comet/issues/1929#issuecomment-3006004580
I don't think this is a bug. Spark likely optimizes the array_contains expression during planning time and replaces it with a boolean literal?
```
checkSparkAnswerAndOperator(spark.sql("SELECT array_contains(cast(array() as array<int>), _2) FROM t1"))
```
See comment https://github.com/apache/datafusion-comet/pull/1923#issuecomment-3002328767 for more information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `array_contains` falls back to Spark in case of Array literal [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead closed issue #1929: `array_contains` falls back to Spark in case of Array literal
URL: https://github.com/apache/datafusion-comet/issues/1929
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `array_contains` falls back to Spark in case of empty Array [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #1929:
URL: https://github.com/apache/datafusion-comet/issues/1929#issuecomment-3006128645
> I don't think this is a bug. Spark likely optimizes the array_contains expression during planning time and replaces it with a boolean literal?
>
> ```
> checkSparkAnswerAndOperator(spark.sql("SELECT array_contains(cast(array() as array<int>), _2) FROM t1"))
> ```
>
> See comment [#1923 (comment)](https://github.com/apache/datafusion-comet/pull/1923#issuecomment-3002328767) for more information
Thanks @andygrove that makes sense. I also checked
```
checkSparkAnswerAndOperator(
spark.sql("SELECT array_contains((CASE WHEN _2 =_3 THEN array() END), _4) FROM t1"));
```
But I still can see the fallback 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `array_contains` falls back to Spark in case of empty Array [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1929:
URL: https://github.com/apache/datafusion-comet/issues/1929#issuecomment-3006195157
@comphead Hmm, do you see a fallback reason?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org