You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2025/08/17 15:20:11 UTC
[I] `array_intersect` test fails when defaul scan is `auto` [datafusion-comet]
andygrove opened a new issue, #2174:
URL: https://github.com/apache/datafusion-comet/issues/2174
### Describe the bug
While working on https://github.com/apache/datafusion-comet/issues/2172, I discovered a potential issue:
```
- array_intersect *** FAILED *** (653 milliseconds)
Results do not match for query:
Timezone: sun.util.calendar.ZoneInfo[id="America/Los_Angeles",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=America/Los_Angeles,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]
Timezone Env:
== Parsed Logical Plan ==
Project [array_intersect(array((cast(_2#433299 as int) * -1)), array(cast(_9#433306 as int), _10#433307)) AS array_intersect(array((_2 * -1)), array(_9, _10))#433366]
+- SubqueryAlias t1
+- View (`t1`, [_1#433298,_2#433299,_3#433300,_4#433301,_5#433302L,_6#433303,_7#433304,_8#433305,_9#433306,_10#433307,_11#433308L,_12#433309,_13#433310,_14#433311,_15#433312,_16#433313,_17#433314,_18#433315,_19#433316,_20#433317,_21#433318,_id#433319])
+- Relation [_1#433298,_2#433299,_3#433300,_4#433301,_5#433302L,_6#433303,_7#433304,_8#433305,_9#433306,_10#433307,_11#433308L,_12#433309,_13#433310,_14#433311,_15#433312,_16#433313,_17#433314,_18#433315,_19#433316,_20#433317,_21#433318,_id#433319] parquet
== Analyzed Logical Plan ==
array_intersect(array((_2 * -1)), array(_9, _10)): array<int>
Project [array_intersect(array((cast(_2#433299 as int) * -1)), array(cast(_9#433306 as int), _10#433307)) AS array_intersect(array((_2 * -1)), array(_9, _10))#433366]
+- SubqueryAlias t1
+- View (`t1`, [_1#433298,_2#433299,_3#433300,_4#433301,_5#433302L,_6#433303,_7#433304,_8#433305,_9#433306,_10#433307,_11#433308L,_12#433309,_13#433310,_14#433311,_15#433312,_16#433313,_17#433314,_18#433315,_19#433316,_20#433317,_21#433318,_id#433319])
+- Relation [_1#433298,_2#433299,_3#433300,_4#433301,_5#433302L,_6#433303,_7#433304,_8#433305,_9#433306,_10#433307,_11#433308L,_12#433309,_13#433310,_14#433311,_15#433312,_16#433313,_17#433314,_18#433315,_19#433316,_20#433317,_21#433318,_id#433319] parquet
== Optimized Logical Plan ==
Project [array_intersect(array((cast(_2#433299 as int) * -1)), array(cast(_9#433306 as int), _10#433307)) AS array_intersect(array((_2 * -1)), array(_9, _10))#433366]
+- Relation [_1#433298,_2#433299,_3#433300,_4#433301,_5#433302L,_6#433303,_7#433304,_8#433305,_9#433306,_10#433307,_11#433308L,_12#433309,_13#433310,_14#433311,_15#433312,_16#433313,_17#433314,_18#433315,_19#433316,_20#433317,_21#433318,_id#433319] parquet
== Physical Plan ==
*(1) CometColumnarToRow
+- CometProject [array_intersect(array((_2 * -1)), array(_9, _10))#433366], [array_intersect(array((cast(_2#433299 as int) * -1)), array(cast(_9#433306 as int), _10#433307)) AS array_intersect(array((_2 * -1)), array(_9, _10))#433366]
+- CometScan [native_iceberg_compat] parquet [_2#433299,_9#433306,_10#433307] Batched: true, DataFilters: [], Format: CometParquet, Location: InMemoryFileIndex(1 paths)[file:/home/andy/git/apache/datafusion-comet/spark/target/tmp/spark-0ed..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<_2:tinyint,_9:smallint,_10:int>
== Results ==
== Results ==
!== Correct Answer - 10000 == == Spark Answer - 10000 ==
struct<array_intersect(array((_2 * -1)), array(_9, _10)):array<int>> struct<array_intersect(array((_2 * -1)), array(_9, _10)):array<int>>
![ArrayBuffer(-1)] [ArrayBuffer()]
![ArrayBuffer(-1)] [ArrayBuffer()]
![ArrayBuffer(-1)] [ArrayBuffer()]
![ArrayBuffer(-1)] [ArrayBuffer()]
![ArrayBuffer(-1)] [ArrayBuffer()]
```
### Steps to reproduce
Update `CometTestBase` to remove the config setting the scan impl to `native_comet`
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `array_intersect` test fails when default scan is `auto` [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich closed issue #2174: `array_intersect` test fails when default scan is `auto`
URL: https://github.com/apache/datafusion-comet/issues/2174
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `array_intersect` test fails when default scan is `auto` [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #2174:
URL: https://github.com/apache/datafusion-comet/issues/2174#issuecomment-3225710788
the issue is likely connected to parquet file reader, I can reproduce with specific parquet created by `makeParquetFileAllPrimitiveTypes`. The same data and data types written by `spark.parquet.write` works fine.
The localized test is
```
test("array_intersect1") {
// TODO test fails if scan is auto
// https://github.com/apache/datafusion-comet/issues/2174
withSQLConf(
CometConf.COMET_NATIVE_SCAN_IMPL.key -> CometConf.SCAN_NATIVE_ICEBERG_COMPAT,
CometConf.COMET_EXPR_ALLOW_INCOMPATIBLE.key -> "true") {
spark.read.parquet("/tmp/test.parquet").createOrReplaceTempView("t1")
sql("select _2, _9, _10 from t1").show(false)
checkSparkAnswerAndOperator(sql("select array(_2), array(_9, _10) from (select _2, _9, _10 from t1)"))
}
}
```
The parquet file attached
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `array_intersect` test fails when default scan is `auto` [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #2174:
URL: https://github.com/apache/datafusion-comet/issues/2174#issuecomment-3225711921
[test.zip](https://github.com/user-attachments/files/21995757/test.zip)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org