You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/07/23 21:31:23 UTC
[I] Native Parquet scan ignores INT96 legacy-calendar rebase, returning silently wrong timestamps [datafusion-comet]
andygrove opened a new issue, #5011:
URL: https://github.com/apache/datafusion-comet/issues/5011
### Describe the bug
The INT96 counterpart of #5010. Comet's native Parquet scan does not apply the hybrid-Julian → proleptic-Gregorian rebase to INT96 timestamp columns, so pre-1582 timestamps read from a legacy-calendar file come back shifted. `spark.sql.parquet.int96RebaseModeInRead` (and the `spark.sql.legacy.parquet.int96RebaseModeInRead` alias) is one of the configs listed under #4180; Comet reads neither, and also ignores the `org.apache.spark.legacyINT96` file metadata that takes precedence over it.
Filed separately from #5010 because the INT96 read path, the metadata key and the governing config are all distinct from the INT32/INT64 date-time path, even though the root cause is the same missing rebase in the native scan.
### Steps to reproduce
```scala
val path = "/tmp/legacy_int96"
spark.conf.set("spark.sql.parquet.outputTimestampType", "INT96")
spark.conf.set("spark.sql.parquet.int96RebaseModeInWrite", "LEGACY")
spark.sql("""SELECT cast(s as timestamp) as ts
FROM VALUES ('1000-01-01 12:34:56'),('1990-01-01 00:00:00') AS v(s)""")
.write.mode("overwrite").parquet(path)
spark.conf.unset("spark.sql.parquet.outputTimestampType")
spark.conf.unset("spark.sql.parquet.int96RebaseModeInWrite")
spark.conf.set("spark.comet.enabled", "false")
spark.read.parquet(path).show(false)
// +---------------------+
// |ts |
// +---------------------+
// |1000-01-01 12:34:56 |
// |1990-01-01 00:00:00 |
// +---------------------+
spark.conf.set("spark.comet.enabled", "true")
spark.read.parquet(path).show(false)
// +---------------------+
// |ts |
// +---------------------+
// |1000-01-06 12:41:58 | <-- wrong
// |1990-01-01 00:00:00 |
// +---------------------+
```
No non-default config is needed on the read side: Spark rebases because the file carries `org.apache.spark.legacyINT96`. Setting `spark.sql.parquet.int96RebaseModeInRead` to `LEGACY`, `CORRECTED` or `EXCEPTION` makes no difference to Comet's output either.
### Expected behavior
Same options as #5010: implement the INT96 rebase honouring the file metadata and `spark.sql.parquet.int96RebaseModeInRead`, or fall back to Spark for affected files and non-default config values.
### Additional context
* Reproduced on `apache/main` @ `0761e549a`, default Maven profile (Spark 4.1.2), macOS aarch64, debug build.
* `spark.comet.scan.enabled=false` makes the results match, isolating this to the native scan.
* Found while auditing the config inventory in #4180.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Native Parquet scan ignores INT96 legacy-calendar rebase, returning silently wrong timestamps [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #5011: Native Parquet scan ignores INT96 legacy-calendar rebase, returning silently wrong timestamps
URL: https://github.com/apache/datafusion-comet/issues/5011
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Native Parquet scan ignores INT96 legacy-calendar rebase, returning silently wrong timestamps [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #5011:
URL: https://github.com/apache/datafusion-comet/issues/5011#issuecomment-5063767232
Closing as a duplicate of #5010.
The INT96 rebase gap has the same root cause (the native scan performs no hybrid-Julian to proleptic-Gregorian rebase) and is covered by the same user-guide text in `compatibility/scans.md`, which speaks of "dates or timestamps" without an INT96 carve-out. Splitting it out added no distinct work item.
The INT96 repro is preserved above, and #5010 now carries the tracking role plus the two parts of this that are not documented: `spark.comet.exceptionOnDatetimeRebase` being dead code, and the documented scope being narrower than the actual behaviour.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org