[jira] [Created] (SPARK-58484) CSV scan fails with ArrayIndexOutOfBoundsException when a pushed filter has no column references and the required schema is empty

["Josh Rosen (Jira)" <ji...@apache.org>]

Josh Rosen created SPARK-58484:
----------------------------------

             Summary: CSV scan fails with ArrayIndexOutOfBoundsException when a pushed filter has no column references and the required schema is empty
                 Key: SPARK-58484
                 URL: https://issues.apache.org/jira/browse/SPARK-58484
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 4.0.0
            Reporter: Josh Rosen


The query
CREATE TABLE m(c0 INT) USING CSV;
INSERT INTO m VALUES (1);
SELECT COUNT(*) FROM m WHERE (SELECT BOOL_OR(true) FROM m);
crashes at execution time with
{code:java}
java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
        at org.apache.spark.sql.catalyst.OrderedFilters.$anonfun$predicates$3(OrderedFilters.scala:63)
        at org.apache.spark.sql.catalyst.OrderedFilters.$anonfun$predicates$3$adapted(OrderedFilters.scala:50)
        at scala.collection.immutable.List.foreach(List.scala:323)
        at org.apache.spark.sql.catalyst.OrderedFilters.<init>(OrderedFilters.scala:50)
        at org.apache.spark.sql.catalyst.csv.UnivocityParser.<init>(UnivocityParser.scala:119)
[...]{code}
The WHERE clause is an uncorrelated boolean scalar subquery referencing no columns.

Since SPARK-43402 such predicates are retained as data filters; at execution time the finished subquery is substituted as {{Literal(true)}} and translated to {{sources.AlwaysTrue}} , which is a filter with no references. 

With {{COUNT(*)}} the required schema is empty. {{}}

{{OrderedFilters}} sizes its predicate-group arrays by {{requiredSchema.fields.length}} (which is 0 in this case) but assigns reference-free filters to index 0, so {{groupedFilters(0)}} throws. The reordering step [immediately below|https://github.com/apache/spark/blob/4dcdd4dba544210898180a410b8b30e0b85157cb/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/OrderedFilters.scala#L65] already guards {{{}len > 0{}}}; the assignment loop does not. {{JsonFilters}} handles the same input by mapping reference-free filters over the (empty) field-name set, which is why JSON does not crash on a similar query.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org