You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "AbhinavBattu (via GitHub)" <gi...@apache.org> on 2026/08/11 19:48:21 UTC
[PR] [SPARK-58484][SQL] Fix ArrayIndexOutOfBoundsException in OrderedFilte… [spark]
AbhinavBattu opened a new pull request, #57936:
URL: https://github.com/apache/spark/pull/57936
### What changes were proposed in this pull request?
`OrderedFilters` groups pushed down filters by the position of the last schema
field they reference, so that a filter can be applied to a partially parsed row
as soon as its referenced fields are set. The group arrays are sized by
`requiredSchema.fields.length`, but filters without references (`AlwaysTrue`,
`AlwaysFalse`) are assigned to index 0. When the required schema has no fields
the arrays are empty and the assignment throws.
This PR skips the grouping when the required schema has no fields. In that case
`skipRow()` can never be invoked, because it asserts that the index points to a
field of the required schema, so there is no position at which such a filter
could be applied.
Note that `StructFilters.pushedFilters()` is correct in keeping these filters:
it checks that a filter does not reference a field missing from the schema, and
a filter without references does not. Reference-free filters are also worth
keeping in general, since a pushed `AlwaysFalse` lets the reader skip rows. The
faulty assumption was in `OrderedFilters`, which required not just that every
reference is present, but that at least one field exists.
`JsonFilters` already handles the same input, because it spreads reference-free
filters over the set of schema field names rather than pinning them to index 0.
With an empty schema that set is empty and the filter is dropped, which is why
JSON does not crash on an equivalent query.
The reordering step immediately below the loop already guarded `len > 0`; the
assignment loop did not.
### Why are the changes needed?
The following query fails at execution time:
```sql
CREATE TABLE m(c0 INT) USING CSV;
INSERT INTO m VALUES (1);
SELECT COUNT(*) FROM m WHERE (SELECT BOOL_OR(true) FROM m);
```
```
java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
at org.apache.spark.sql.catalyst.OrderedFilters.$anonfun$predicates$3(OrderedFilters.scala:63)
at org.apache.spark.sql.catalyst.OrderedFilters.<init>(OrderedFilters.scala:50)
at org.apache.spark.sql.catalyst.csv.UnivocityParser.<init>(UnivocityParser.scala:119)
```
Through the SQL path the exception surfaces wrapped as
`[FAILED_READ_FILE.NO_HINT]` with the `ArrayIndexOutOfBoundsException` as its
cause.
The `WHERE` clause is an uncorrelated boolean scalar subquery referencing no
columns. Since SPARK-43402 such predicates are retained as data filters; at
execution the finished subquery is substituted as `Literal(true)` and translated
to `sources.AlwaysTrue`, a filter with no references. With `COUNT(*)` the
required schema is empty, so both conditions hold at once.
`OrderedFilters` is also used by the Avro reader, which is affected in the same
way and is fixed by the same change.
### Does this PR introduce _any_ user-facing change?
Yes. A query that pushes down a filter without column references while requiring
no columns previously failed with `ArrayIndexOutOfBoundsException` and now
returns the correct result.
### How was this patch tested?
Two new tests, both confirmed to fail before this change and pass after it.
`StructFiltersSuite` is shared by `OrderedFiltersSuite` and `JsonFiltersSuite`,
so the unit test covers both implementations and shows that JSON was already
correct. The test asserts only that building the filters succeeds, because with
an empty required schema there is nothing observable to assert: the predicates
are private and `skipRow()` cannot be called at all.
```
build/sbt 'catalyst/testOnly *OrderedFiltersSuite *JsonFiltersSuite'
```
An end to end test was added to `CSVSuite`, which is inherited by `CSVv1Suite`
and `CSVv2Suite`, so the query from the JIRA is covered on both the DSv1 and
DSv2 reader paths and its result is checked.
```
build/sbt 'sql/testOnly *CSVv1Suite *CSVv2Suite -- -z "SPARK-58484"'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58484][SQL] Fix ArrayIndexOutOfBoundsException in OrderedFilters when the required schema is empty [spark]
Posted by "AbhinavBattu (via GitHub)" <gi...@apache.org>.
AbhinavBattu commented on PR #57936:
URL: https://github.com/apache/spark/pull/57936#issuecomment-5264156083
Thanks for the reviews @uros-b @yyanyy!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58484][SQL] Fix ArrayIndexOutOfBoundsException in OrderedFilters when the required schema is empty [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #57936:
URL: https://github.com/apache/spark/pull/57936#issuecomment-5262867937
Thank you @AbhinavBattu and @yyanyy!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org