[jira] [Created] (SPARK-59375) flatten raises an internal error instead of COLLECTION_SIZE_LIMIT_EXCEEDED in codegen

["sepuri sai krishna (Jira)" <ji...@apache.org>]

sepuri sai krishna created SPARK-59375:
------------------------------------------

             Summary: flatten raises an internal error instead of COLLECTION_SIZE_LIMIT_EXCEEDED in codegen
                 Key: SPARK-59375
                 URL: https://issues.apache.org/jira/browse/SPARK-59375
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 4.0.4, 4.1.3, 4.2.0, 4.3.0
            Reporter: sepuri sai krishna


{{Flatten.genCodeForNumberOfElements}} does not check the result element count
against {{ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH}}, while the interpreted path
in {{Flatten.nullSafeEval}} does. The two paths therefore report different error
conditions for the same input: the interpreted path raises
{{COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION}}, and codegen reaches
{{ArrayData.allocateArrayData}} and raises the internal {{_LEGACY_ERROR_TEMP_2176}}.

Same pattern as SPARK-58631, which fixed it for {{array_repeat}}.

Priority is Minor because the limit is not cheaply reachable here: {{flatten}}
needs its input arrays to hold that many elements for real, so an executor will
normally run out of memory first.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org