[jira] [Created] (SPARK-60268) Fail window partitions larger than Int.MaxValue rows while buffering instead of after

["Mark Jarvin (Jira)" <ji...@apache.org>]

Mark Jarvin created SPARK-60268:
-----------------------------------

             Summary: Fail window partitions larger than Int.MaxValue rows while buffering instead of after
                 Key: SPARK-60268
                 URL: https://issues.apache.org/jira/browse/SPARK-60268
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Mark Jarvin


h2. Summary

Since SPARK-59929, window execution rejects a partition with more than {{Int.MaxValue}} rows when any of its frames cannot handle it, using error {{WINDOW_FUNCTION_PARTITION_SIZE_EXCEEDS_LIMIT}}. The check ({{WindowEvaluatorFactoryBase.checkPartitionSizeLimit}}) runs only after the whole partition has been read and buffered/spilled, in {{WindowEvaluatorFactory.fetchNextPartition}} and {{ArrowWindowPythonEvaluatorFactory.fetchNextPartition}}.

So a query that is going to fail still reads and spills the entire oversized partition first. The reproduction in SPARK-59929 took about 30 minutes over 2.2B rows on one core, plus the disk used by the spill, before the error appeared.

h2. Proposal

When any frame does not support large partitions, check the buffered row count while rows are being added (one {{Long}} comparison per row) and throw as soon as it exceeds {{Int.MaxValue}}. Frames that do support large partitions (LEAD/LAG and NTH_VALUE over the whole partition, see SPARK-59941) must keep buffering.

Needs a test with a stubbed or small buffer that reports a large length, since a real oversized partition is impractical in a unit test.

Depends on SPARK-59941 (introduces {{WindowFunctionFrame.supportsLargePartition}}).




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org