Re: [PR] feat: support ANSI interval types in native Parquet scans [datafusion-comet]

["sunchao (via GitHub)" <gi...@apache.org>]

sunchao commented on code in PR #5161:
URL: https://github.com/apache/datafusion-comet/pull/5161#discussion_r4104921866


##########
spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala:
##########
@@ -1117,6 +1117,10 @@ case class CometScanTypeChecker() extends DataTypeSupport with CometTypeShim {
         false
       case s: StructType if s.fields.isEmpty =>
         false
+      case _: YearMonthIntervalType | _: DayTimeIntervalType =>

Review Comment:
   [P2] Update the existing interval fallback fixtures alongside these support gates. `expressions/aggregate/collect_list.sql:292,295` still requires interval scans to fall back, and `expressions/math/abs.sql:62` still requires a null year-month literal to fall back. These queries now execute through Comet, so `CometSqlFileTestSuite` fails instead of accepting the newly supported behavior. This breaks every Spark-version expression job and the required CI gate. Change those records to normal `query` assertions, update their obsolete comments, and rerun both fixtures.
   
   Evidence: Exact-head run 34703461345 reproduces this on Spark 3.4, 3.5, 4.0, 4.1 and 4.2. Each job reports three failures: `abs.sql` and both dictionary configurations of `collect_list.sql`. Spark 4.1 job 103583366054 reports `Expected fallback reason 'Unsupported data type YearMonthIntervalType' but no fallback reasons were found` for `SELECT abs(CAST(NULL AS INTERVAL YEAR TO MONTH))`, and the analogous missing `Unsupported ym of type YearMonthIntervalType` reason for `collect_list(ym)`. Log: https://github.com/apache/datafusion-comet/actions/runs/34703461345/job/103583366054



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org