You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/10/15 21:46:17 UTC
[I] Native Parquet scan reads TIMESTAMP(NANOS) columns as timestamp where Spark rejects them [datafusion-comet]
andygrove opened a new issue, #7034:
URL: https://github.com/apache/datafusion-comet/issues/7034
### Describe the bug
The native Parquet scan reads an INT64 `TIMESTAMP(NANOS)` column through a schema that declares `timestamp`, and on Spark 4.x one that declares `timestamp_ntz`, and returns the values truncated to microseconds. Spark has no reader for that combination and raises `[FAILED_READ_FILE.PARQUET_COLUMN_DATA_TYPE_MISMATCH]`. It applies to top-level and nested columns. pyarrow 13 and later write nanosecond timestamps by default, so a catalog table declared over pyarrow output reaches this.
`spark_has_updater` (`native/core/src/parquet/schema_adapter.rs` around lines 553-713) mirrors `ParquetVectorUpdaterFactory.getUpdater` for the conversions it checks, but has no rule that rejects a nanosecond timestamp read as a microsecond Spark timestamp, so the adapter casts it. Spark reads `TIMESTAMP(NANOS)` only as `bigint`, with `spark.sql.legacy.parquet.nanosAsLong=true`, and Comet matches that case.
### Steps to reproduce
Write a file with parquet-mr's `RecordConsumer` (or pyarrow) holding `optional int64 t (TIMESTAMP(NANOS,true))` and `optional group s { optional int64 t (TIMESTAMP(NANOS,true)); }`, with one row whose values are `1600000000123456789`. Then, with `spark.sql.session.timeZone=UTC`:
```scala
spark.read.schema("t timestamp").parquet(path).collect()
spark.read.schema("s struct<t: timestamp>").parquet(path).collect()
spark.read.schema("t timestamp_ntz").parquet(path).collect()
```
On `main` at 66c63bd307:
| Read schema | Spark 3.5.9 and 4.1.3 | Comet native scan |
| --- | --- | --- |
| `t timestamp` | `PARQUET_COLUMN_DATA_TYPE_MISMATCH ... column: [t], physicalType: INT64, logicalType: timestamp` | `2020-09-13 12:26:40.123456` UTC |
| `s struct<t: timestamp>` | the same error for `[s, t]` | `{2020-09-13 12:26:40.123456}` |
| `t timestamp_ntz` | the same error | 4.1.3: `2020-09-13T12:26:40.123456`. 3.5.9: the same error as Spark |
### Expected behavior
The native scan rejects a nanosecond timestamp read as `timestamp` or `timestamp_ntz` the way Spark does, deferred to decode time like the other updater rejections (`RejectOnNonEmpty`).
### Additional context
Found in a nested-type audit of the native scans. Comet returns plausible values where Spark fails, so this is a compatibility gap rather than a wrong result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Native Parquet scan reads TIMESTAMP(NANOS) columns as timestamp where Spark rejects them [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #7034: Native Parquet scan reads TIMESTAMP(NANOS) columns as timestamp where Spark rejects them
URL: https://github.com/apache/datafusion-comet/issues/7034
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org