You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org> on 2026/10/18 10:27:30 UTC
[PR] [SPARK-60326][SQL] Pass the ORC SearchArgument to the reader directly [spark]
SEPURI-SAI-KRISHNA opened a new pull request, #59584:
URL: https://github.com/apache/spark/pull/59584
### What changes were proposed in this pull request?
The native ORC readers (V1 `OrcFileFormat` and V2 `OrcPartitionReaderFactory`) now pass the pushed `SearchArgument` to the ORC reader with `Reader.Options.searchArgument`, instead of storing it in the Hadoop conf with `OrcInputFormat.setSearchArgument` for `OrcInputFormat.buildOptions` to decode. `OrcColumnarBatchReader.initialize` and `OrcUtils.createOrcMapreduceRecordReader` take the `SearchArgument` as a parameter for this.
### Why are the changes needed?
ORC filter pushdown drops rows when a TIMESTAMP filter literal has sub-millisecond digits. With the default configuration, on 4.0.3, 4.2.0 (official binaries) and master:
```sql
CREATE TABLE t USING orc AS SELECT 1 AS id, TIMESTAMP'2024-06-01 12:00:00.123456' AS ts;
SELECT count(*) FROM t WHERE ts = TIMESTAMP'2024-06-01 12:00:00.123456'; -- 0, expected 1
SELECT count(*) FROM t WHERE ts <= TIMESTAMP'2024-06-01 12:00:00.123456'; -- 0, expected 1
CREATE TABLE m USING orc AS SELECT id, TIMESTAMP'2024-06-01 12:00:00.123' AS ts FROM range(10);
SELECT count(*) FROM m WHERE ts != TIMESTAMP'2024-06-01 12:00:00.123456'; -- 0, expected 10
SELECT count(*) FROM m WHERE ts < TIMESTAMP'2024-06-01 12:00:00.123001'; -- 0, expected 10
```
The same happens with `<=>` and `IN`, with the V1 and V2 readers, and with the vectorized reader on and off. All are correct with `spark.sql.orc.filterPushdown=false`.
The `SearchArgument` built by `OrcFilters` has the exact literal, and the ORC statistics are exact too. But `OrcInputFormat.setSearchArgument` serializes the `SearchArgument` with a plain `Kryo`, whose serializer for `java.sql.Timestamp` only keeps milliseconds, so the reader evaluates `ts = 12:00:00.123`, `ts <= 12:00:00.123` or `ts != 12:00:00.123` against the statistics and skips stripes and row groups that hold matching rows. `>` and `>=` only get wider. It shows when the literal is within a millisecond of the minimum or maximum of a row group, stripe or file.
The `SearchArgument` is built on the executor for each file, right before the reader is created, so there is no need to go through the conf. This also means the readers no longer write it into the broadcast Hadoop conf that the tasks of an executor share.
The nanosecond timestamp types (SPARK-57823) are pushed down the same way and lost rows the same way, e.g. `ts = <TIMESTAMP_NTZ(9) value with sub-ms digits>`; they are fixed by this change too.
`spark.sql.orc.impl=hive` has the same problem, but there Hive's `OrcInputFormat` reads the `SearchArgument` from the conf (`sarg.pushdown`) with a plain `Kryo` as well, so it cannot be fixed this way. It will be handled separately.
### Does this PR introduce _any_ user-facing change?
Yes. Queries on ORC files with a TIMESTAMP filter whose literal has sub-millisecond digits now return the rows that were dropped before. Pushdown still skips the same stripes and row groups otherwise.
### How was this patch tested?
New test in `OrcQuerySuite` (run by `OrcV1QuerySuite` and `OrcV2QuerySuite`): `=`, `<=>`, `<=`, `<`, `IN`, `!=` and `>=` with sub-millisecond literals on files of one value each, with the vectorized reader on and off, and checks that the files that do not match are still skipped. It fails without this change.
A separate matrix compared pushdown on and off for these predicates and more (`BETWEEN`, `NOT`, `OR`, `NOT IN`), on a file of three row groups, a file of values on a millisecond, and values in 1900, 1969 and 9999, for V1 and V2, the vectorized reader on and off, and the UTC and America/Los_Angeles time zones: 20 of 176 cases returned fewer rows with pushdown before this change, none after, and the number of rows read from the files is the same before and after in all other cases. The `=` and `<=` cases of the nanosecond types, for V1 and V2, also return the right rows now.
These suites pass: `OrcV1QuerySuite`, `OrcV2QuerySuite`, `OrcFilterSuite`, `OrcV1FilterSuite`, `OrcSourceV1Suite`, `OrcSourceV2Suite`, `OrcColumnarBatchReaderSuite`, `OrcPartitionReaderFactorySuite`, `OrcPartitionDiscoverySuite`, `OrcV1PartitionDiscoverySuite`, `OrcV1SchemaPruningSuite`, `OrcV2SchemaPruningSuite` and `OrcEncryptionSuite` in `sql/core`, and `HiveOrcQuerySuite`, `HiveOrcSourceSuite`, `HiveOrcPartitionDiscoverySuite`, `OrcHadoopFsRelationSuite` and `HiveOrcHadoopFsRelationSuite` in `sql/hive`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60326][SQL] Pass the ORC SearchArgument to the reader directly [spark]
Posted by "spark-test (via GitHub)" <gi...@apache.org>.
spark-test commented on PR #59584:
URL: https://github.com/apache/spark/pull/59584#issuecomment-6259792996
The AI review requested by @HyukjinKwon finished.
*Generated by Omnigent on Databricks*
<!-- SPARK_OSS_REVIEW_REQUEST 33605593260 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60326][SQL] Pass the ORC SearchArgument to the reader directly [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #59584:
URL: https://github.com/apache/spark/pull/59584#issuecomment-6269899346
LGTM, thank you @SEPURI-SAI-KRISHNA!
cc @cloud-fan and @LuciferYang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60326][SQL] Pass the ORC SearchArgument to the reader directly [spark]
Posted by "SEPURI-SAI-KRISHNA (via GitHub) via reviews" <re...@spark.apache.org>.
SEPURI-SAI-KRISHNA commented on PR #59584:
URL: https://github.com/apache/spark/pull/59584#issuecomment-6624858823
Closing this one: the same fix was merged in #59682 (SPARK-60436), which passes the `SearchArgument` to the ORC reader directly in the same way, and SPARK-60396 / SPARK-60423 handled the Hive reader. Thanks @uros-b for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60326][SQL] Pass the ORC SearchArgument to the reader directly [spark]
Posted by "SEPURI-SAI-KRISHNA (via GitHub) via reviews" <re...@spark.apache.org>.
SEPURI-SAI-KRISHNA closed pull request #59584: [SPARK-60326][SQL] Pass the ORC SearchArgument to the reader directly
URL: https://github.com/apache/spark/pull/59584
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org