You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2026/09/18 20:18:04 UTC
[I] Support pruning on INT96 timestamp columns via `SortOrder::INT96_TIMESTAMP` / `ColumnOrder::INT96_TIMESTAMP_ORDER` [datafusion]
alamb opened a new issue, #25484:
URL: https://github.com/apache/datafusion/issues/25484
### Is your feature request related to a problem or challenge?
The Parquet specification historically left the sort order of `INT96` undefined, so DataFusion doesn't trust `min`/`max` statistics of `INT96` columns (TODO code ref) and therefore cannot prune row groups, pages, or files using predicates on those columns.
However, Parquet files written by Spark, Hive, Impala, and Databricks Photon commonly store timestamps as the (deprecated) `INT96` physical type and show no signs of changing this behavior. :thumbsdown:
This means any query with a timestamp filter over a Spark-written dataset scans every row group, for example, using this file; TODO FILE
```sql
````
Note there is no pruning
I made the file like this
<details>
```shell
...
```
</details>
The Parquet specification recently added a new `ColumnOrder` variant, `INT96_TIMESTAMP_ORDER`, in [apache/parquet-format#584]: https://github.com/apache/parquet-format/blob/2076361bb64e2de9ca6a8d06eda025a6fa4e9df6/src/main/thrift/parquet.thrift#L1230-L1233 to explicitly define the order
arrow-rs / parquet `60.0.0` implemented this in
- https://github.com/apache/arrow-rs/pull/10106
However, to avoid introducing any new correctness bugs, when upgrading to arrow/parquet `60.0.0` in https://github.com/apache/datafusion/pull/25335, I changed `has_untrusted_min_max_order` to keep treating `INT96` columns as untrusted (see the review comment at https://github.com/apache/datafusion/pull/25335/changes#r4018522067 and the code)
However, this means that DataFusion will *still* not prune files with `INT96` , even those that could
- File-level statistics: https://github.com/apache/datafusion/blob/5a502dffce2f1881479e8c1a5b68a4a5108e7934/datafusion/datasource-parquet/src/metadata.rs#L555-L585
- Row group pruning: https://github.com/apache/datafusion/blob/5a502dffce2f1881479e8c1a5b68a4a5108e7934/datafusion/datasource-parquet/src/row_group_filter.rs#L518-L531
- Page pruning: https://github.com/apache/datafusion/blob/5a502dffce2f1881479e8c1a5b68a4a5108e7934/datafusion/datasource-parquet/src/page_filter.rs#L760-L764
### Describe the solution you'd like
Trust `INT96` min/max statistics when column actually has `ColumnOrder::INT96_TIMESTAMP_ORDER`, and use them for file, row group, and page pruning.
### Describe alternatives you've considered
N/A
### Additional context
- Upgrade PR where the conservative check was added: https://github.com/apache/datafusion/pull/25335
- arrow-rs implementation: https://github.com/apache/arrow-rs/pull/10106
- Parquet format change: https://github.com/apache/parquet-format/pull/584
- Prior arrow-rs discussion of incorrect `INT96` statistics: https://github.com/apache/arrow-rs/issues/7686
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support pruning on INT96 timestamp columns via `SortOrder::INT96_TIMESTAMP` / `ColumnOrder::INT96_TIMESTAMP_ORDER` [datafusion]
Posted by "Seanium (via GitHub)" <gi...@apache.org>.
Seanium commented on issue #25484:
URL: https://github.com/apache/datafusion/issues/25484#issuecomment-5966023149
Implemented in draft #25987, with the required Arrow statistics support in apache/arrow-rs#11348. Tests cover file, row-group, page and runtime pruning, configured timestamp units, legacy ordering and overflow fallback. The DataFusion draft remains blocked on the Arrow dependency; files without explicit `INT96_TIMESTAMP_ORDER` remain untrusted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support pruning on INT96 timestamp columns via `SortOrder::INT96_TIMESTAMP` / `ColumnOrder::INT96_TIMESTAMP_ORDER` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #25484:
URL: https://github.com/apache/datafusion/issues/25484#issuecomment-5735749741
Note I think we will have to add support for Int96 timestamp statistics in the arrow-rs StatisticsConverter now too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support pruning on INT96 timestamp columns via `SortOrder::INT96_TIMESTAMP` / `ColumnOrder::INT96_TIMESTAMP_ORDER` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #25484:
URL: https://github.com/apache/datafusion/issues/25484#issuecomment-5735780447
Two things I found while building the reproducer that affect how this should be implemented.
## 1. Relaxing `has_untrusted_min_max_order` is necessary but not sufficient
`StatisticsConverter` never surfaces INT96 bounds, so the pruning predicate sees nulls even when the footer holds correct statistics. Against the file attached above:
```
arrow ts field : Field { "ts": Timestamp(ns) }
row_group_mins : PrimitiveArray<Timestamp(ns)> [ null, null ]
row_group_maxes: PrimitiveArray<Timestamp(ns)> [ null, null ]
=> 2 of 2 row-group mins came back NULL
```
The converter's `DataType::Timestamp` arm only reads `Statistics::Int64`, so `Statistics::Int96` falls through as null:
https://github.com/apache/arrow-rs/blob/60.0.0/parquet/src/arrow/arrow_reader/statistics.rs#L476-L484
So this needs an arrow-rs change as well, converting `Statistics::Int96` into the timestamp type the reader produces (`Timestamp(Nanosecond)` by default, or the unit chosen by `datafusion.execution.parquet.coerce_int96`). Note that `Int96::to_nanos` wraps on overflow, so an out-of-range Julian day should yield a null bound rather than a silently wrong one.
## 2. No Spark-written file can benefit from this yet
parquet-java 1.17's `PrimitiveType.requireValidColumnOrder` rejects every `ColumnOrder` except `UNDEFINED` for INT96 ("The column order %s is not supported by INT96"), and an undefined order means no statistics are written at all. I confirmed this against the jars Spark 4.2 ships: a Spark-written INT96 column has no min/max whatsoever, which is why the example above is written with parquet-rs rather than Spark.
So the immediate beneficiaries are files written by parquet-rs 60+. Spark, Hive, Impala and Photon files follow only once parquet-java also implements [parquet-format#584](https://github.com/apache/parquet-format/pull/584). That does not change the case for doing this, but it is worth being clear that it will not retroactively speed up the existing INT96 corpus.
## Note for whoever picks this up
`undefined_int96_order_is_never_trusted` in `datafusion/datasource-parquet/src/statistics_order_tests.rs` currently asserts the opposite of the desired behaviour, including for the `ColumnOrder::INT96_TIMESTAMP_ORDER` case, and will need updating.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org