You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "MaxGekk (via GitHub)" <gi...@apache.org> on 2026/06/26 18:40:22 UTC
[PR] [SPARK-57571][SQL] Support TIME predicate pushdown to ORC [spark]
MaxGekk opened a new pull request, #56823:
URL: https://github.com/apache/spark/pull/56823
### What changes were proposed in this pull request?
This PR adds predicate pushdown for the `TIME` (`TimeType`) data type to the ORC reader, matching the existing Parquet TIME filter pushdown (SPARK-51687).
Changes:
- `OrcFilters`:
- `getPredicateLeafType` now maps `TimeType` to `PredicateLeaf.Type.LONG`.
- `castLiteralValue` converts a `java.time.LocalTime` literal to nanos-of-day via `DateTimeUtils.localTimeToNanos`, which is how ORC physically stores `TIME` values (ORC `LONG`, see `OrcSerializer`/`OrcUtils`).
- The existing leaf handlers (`EqualTo`, `EqualNullSafe`, `LessThan`, `LessThanOrEqual`, `GreaterThan`, `GreaterThanOrEqual`, `In`, `IsNull`) then work for `TIME` unchanged.
- Add a `localTimeToLiteral` implicit to the Catalyst `dsl` package, mirroring the existing `timestampNTZToLiteral`, so `TIME` literals can be used in dsl predicates (used by the test).
- Add a test in `OrcFilterSuite` (also exercised by `OrcV1FilterSuite`).
### Why are the changes needed?
Native ORC read/write for `TIME` is already supported (SPARK-54472), but `OrcFilters.getPredicateLeafType` had no `TimeType` case, so `TIME` predicates were not pushed down to the ORC reader. This closes that gap and brings ORC to parity with Parquet, avoiding unnecessary row scans for filters on `TIME` columns.
### Does this PR introduce _any_ user-facing change?
No behavior change in query results. Filters on `TIME` columns over ORC are now pushed down to the reader, which can improve scan performance.
### How was this patch tested?
Added `SPARK-57571: filter pushdown - time` in `OrcFilterSuite`, covering EQ / NULL_SAFE_EQUALS / LT / LTE / GT / GTE / IsNull in both `col op literal` and `literal op col` forms. Ran:
```
build/sbt 'sql/testOnly *OrcFilterSuite *OrcV1FilterSuite'
```
All 40 tests passed. Also ran `./dev/scalastyle` (passed) and scalafmt on the connect/api modules.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor (Claude Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57571][SQL] Support TIME predicate pushdown to ORC [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk closed pull request #56823: [SPARK-57571][SQL] Support TIME predicate pushdown to ORC
URL: https://github.com/apache/spark/pull/56823
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57571][SQL] Support TIME predicate pushdown to ORC [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on PR #56823:
URL: https://github.com/apache/spark/pull/56823#issuecomment-4813620800
Merging to master/4.x. Thank you, @dongjoon-hyun for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org